[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:35 dist-test-slave-f1t5 krb5kdc[7536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:35 dist-test-slave-f1t5 krb5kdc[7536](info): set up 2 sockets
Jun 19 11:09:35 dist-test-slave-f1t5 krb5kdc[7536](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 11:09:36.902235 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:36 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:37.014894 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:42389
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:42389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:37.119639 7549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:37.119928 7549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:37.120005 7549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:37.123991 7549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:37.124073 7549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:37.124111 7549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:37.124171 7549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:37.124210 7549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:37.129272 7549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:42389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:42389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:37.130307 7549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:37.131278 7549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:37.139652 7554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.139645 7557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.139645 7555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:37.139709 7549 server_base.cc:1034] running on GCE node
I20260619 11:09:37.140451 7549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:37.141440 7549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:37.142593 7549 hybrid_clock.cc:648] HybridClock initialized: now 1781867377142569 us; error 36 us; skew 500 ppm
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:37.145190 7549 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:37.146018 7549 webserver.cc:460] Webserver started at http://127.7.90.126:34033/ using document root <none> and password file <none>
I20260619 11:09:37.146507 7549 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:37.146561 7549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:37.146716 7549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:37.148586 7549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "ef44b78af76f943ad11effb04ae9ad30"
server_key_iv: "2170c2bf9272278d7d5b6170dd36b1e3"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.149070 7549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "ef44b78af76f943ad11effb04ae9ad30"
server_key_iv: "2170c2bf9272278d7d5b6170dd36b1e3"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.152495 7549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:09:37.154702 7564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.155861 7549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:37.156002 7549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "ef44b78af76f943ad11effb04ae9ad30"
server_key_iv: "2170c2bf9272278d7d5b6170dd36b1e3"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.156108 7549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:37.180534 7549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:37.183202 7549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:37.183399 7549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:37.190433 7549 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:42389
I20260619 11:09:37.190454 7616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:42389 every 8 connection(s)
I20260619 11:09:37.191321 7549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:37.193953 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:37.199051 7617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48: Bootstrap starting.
I20260619 11:09:37.200431 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 7549
I20260619 11:09:37.200544 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:37.201352 7617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:37.201866 7529 external_mini_cluster.cc:1428] Setting key c56e9da0dd45be10fb34d59a60c3871a
I20260619 11:09:37.202128 7617 log.cc:824] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:37.203895 7617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48: No bootstrap required, opened a new log
I20260619 11:09:37.208477 7617 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } }
I20260619 11:09:37.208710 7617 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:37.208806 7617 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc7b5dd4bd2491a9e7e5c62630cce48, State: Initialized, Role: FOLLOWER
I20260619 11:09:37.209260 7617 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } }
I20260619 11:09:37.209435 7617 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:37.209520 7617 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:37.209632 7617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:37.210451 7617 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } }
I20260619 11:09:37.210770 7617 leader_election.cc:302] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc7b5dd4bd2491a9e7e5c62630cce48; no voters:
I20260619 11:09:37.211068 7617 leader_election.cc:288] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:37.211166 7622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:37.211386 7622 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [term 1 LEADER]: Becoming Leader. State: Replica: cbc7b5dd4bd2491a9e7e5c62630cce48, State: Running, Role: LEADER
I20260619 11:09:37.211675 7622 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } }
I20260619 11:09:37.212206 7617 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:37.212920 7624 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc7b5dd4bd2491a9e7e5c62630cce48. Latest consensus state: current_term: 1 leader_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } } }
I20260619 11:09:37.213055 7624 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:37.213476 7623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc7b5dd4bd2491a9e7e5c62630cce48" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 42389 } } }
I20260619 11:09:37.213663 7623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:37.213982 7631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:37.217226 7631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:37.222769 7631 catalog_manager.cc:1347] Generated new cluster ID: 42b7f40b46224ab7815684c8fd54afa5
I20260619 11:09:37.222832 7631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:37.226361 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.204642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.7.90.126:42389)
0619 11:09:37.205013 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.205027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.206324 (+ 1297us) server_negotiation.cc:407] Connection header received
0619 11:09:37.208843 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.208860 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.209280 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.209684 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.212196 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.213855 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.215479 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.215955 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.216078 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.220298 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.220304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.220329 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.220359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.222204 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.222846 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.222854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.222860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.222985 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.223375 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.223379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.223380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.223950 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.224154 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.224731 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.225004 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260619 11:09:37.234179 7631 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:37.235044 7631 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:37.241412 7631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc7b5dd4bd2491a9e7e5c62630cce48: Generated new TSK 0
I20260619 11:09:37.242029 7631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:37.272696 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:37.383508 7643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:37.383735 7643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:37.383776 7643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:37.387593 7643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:37.387637 7643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:37.387691 7643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:37.392560 7643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:37.393452 7643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:37.394484 7643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:37.404067 7651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.404062 7648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.404119 7649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:37.404539 7643 server_base.cc:1034] running on GCE node
I20260619 11:09:37.404989 7643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:37.405642 7643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:37.406816 7643 hybrid_clock.cc:648] HybridClock initialized: now 1781867377406782 us; error 49 us; skew 500 ppm
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:37.409474 7643 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:37.410339 7643 webserver.cc:460] Webserver started at http://127.7.90.65:45697/ using document root <none> and password file <none>
I20260619 11:09:37.410868 7643 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:37.410959 7643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:37.411192 7643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:37.413281 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "b372b06b484f4955acca5763dd3579cb"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "b6265ed50107d86b5919d2c8d764dd7b"
server_key_iv: "afb315d6d155fc541bebaf90a8293ec6"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.413938 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "b372b06b484f4955acca5763dd3579cb"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "b6265ed50107d86b5919d2c8d764dd7b"
server_key_iv: "afb315d6d155fc541bebaf90a8293ec6"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.417258 7643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:37.419497 7658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.420650 7643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:37.420735 7643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "b372b06b484f4955acca5763dd3579cb"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "b6265ed50107d86b5919d2c8d764dd7b"
server_key_iv: "afb315d6d155fc541bebaf90a8293ec6"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.420809 7643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:37.432870 7643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:37.435616 7643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:37.435752 7643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:37.436213 7643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:37.436913 7643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:37.436959 7643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.436995 7643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:37.437011 7643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.447117 7643 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:37123
I20260619 11:09:37.447139 7771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:37123 every 8 connection(s)
I20260619 11:09:37.447956 7643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:37.448792 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 7643
I20260619 11:09:37.448875 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:37.449172 7529 external_mini_cluster.cc:1428] Setting key 9c0c74ff2b2df2417333f8e2fd4ef751
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:37.466097 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.449748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:48829 (local address 127.7.90.126:42389)
0619 11:09:37.450063 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.450067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.450741 (+ 674us) server_negotiation.cc:407] Connection header received
0619 11:09:37.452134 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.452137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.452180 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.452248 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.457706 (+ 5458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.458309 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.459621 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.459845 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.459964 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.462507 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.462510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.462512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.462535 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.463904 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.464497 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.464500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.464501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.464578 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.464970 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.464972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.464974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.465138 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.465212 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.465876 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.465980 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260619 11:09:37.467351 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.450128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:42389 (local address 127.7.90.65:48829)
0619 11:09:37.450605 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:37.450658 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:37.451831 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:37.452427 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:37.452438 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:37.452936 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:37.453476 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.453489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.458438 (+ 4949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.458445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.459488 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.459497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.460055 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.460062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.460278 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.460874 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:37.460903 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:37.462333 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:37.464019 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.464027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.464045 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.464394 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.464672 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.464678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.464683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.464863 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.465245 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:37.465254 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:37.465616 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 11:09:37.465840 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 11:09:37.466210 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260619 11:09:37.468608 7772 heartbeater.cc:347] Connected to a master server at 127.7.90.126:42389
I20260619 11:09:37.468839 7772 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:37.469300 7772 heartbeater.cc:511] Master 127.7.90.126:42389 requested a full tablet report, sending...
I20260619 11:09:37.470788 7581 ts_manager.cc:194] Registered new tserver with Master: b372b06b484f4955acca5763dd3579cb (127.7.90.65:37123)
I20260619 11:09:37.472136 7581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:48829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:37.476423 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:37.582592 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:37.582844 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:37.582922 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:37.586886 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:37.586937 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:37.587041 7777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:37.592125 7777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:37.593111 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:37.594134 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:37.603477 7783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:37.603525 7777 server_base.cc:1034] running on GCE node
W20260619 11:09:37.603480 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.603478 7785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:37.604058 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:37.604576 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:37.605760 7777 hybrid_clock.cc:648] HybridClock initialized: now 1781867377605694 us; error 49 us; skew 500 ppm
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:37.608469 7777 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:37.609282 7777 webserver.cc:460] Webserver started at http://127.7.90.66:46665/ using document root <none> and password file <none>
I20260619 11:09:37.609870 7777 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:37.609930 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:37.610147 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:37.612116 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "297774e99c8b4a8b8e5accac37cb6835"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "d493e6fa030add22f287d6580811e265"
server_key_iv: "710d545ec805de7bd8f2b26c2f994e1c"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.612668 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "297774e99c8b4a8b8e5accac37cb6835"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "d493e6fa030add22f287d6580811e265"
server_key_iv: "710d545ec805de7bd8f2b26c2f994e1c"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.616374 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:37.618665 7792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.619861 7777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:37.619951 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "297774e99c8b4a8b8e5accac37cb6835"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "d493e6fa030add22f287d6580811e265"
server_key_iv: "710d545ec805de7bd8f2b26c2f994e1c"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.620090 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:37.650671 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:37.653848 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:37.654045 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:37.654628 7777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:37.655412 7777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:37.655457 7777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.655524 7777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:37.655562 7777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.664664 7777 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:33809
I20260619 11:09:37.664705 7905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:33809 every 8 connection(s)
I20260619 11:09:37.665596 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:37.672498 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 7777
I20260619 11:09:37.672628 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:37.672921 7529 external_mini_cluster.cc:1428] Setting key feb9ccd02920f708d8adfc72223bc84f
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:37.679356 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.667324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:53739 (local address 127.7.90.126:42389)
0619 11:09:37.667458 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.667462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.668309 (+ 847us) server_negotiation.cc:407] Connection header received
0619 11:09:37.669696 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.669699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.669754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.669868 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.671003 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.671513 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.672630 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.672931 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.673054 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.675607 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.675611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.675614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.675644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.677272 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.677875 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.677879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.677881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.677964 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.678293 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.678296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.678298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.678471 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.678573 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.679082 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.679194 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:09:37.680346 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.667708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:42389 (local address 127.7.90.66:53739)
0619 11:09:37.668158 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:37.668200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:37.669433 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:37.670049 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:37.670057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:37.670414 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:37.670867 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.670879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.671631 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.671634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.672507 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.672516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.673091 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.673095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.673267 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.673896 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:37.673913 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:37.675450 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:37.677387 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.677393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.677408 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.677755 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.678053 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.678058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.678060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.678195 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.678571 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:37.678578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:37.678853 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 11:09:37.679033 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 11:09:37.679350 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
I20260619 11:09:37.681579 7906 heartbeater.cc:347] Connected to a master server at 127.7.90.126:42389
I20260619 11:09:37.681862 7906 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:37.682330 7906 heartbeater.cc:511] Master 127.7.90.126:42389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:09:37.683429 7581 ts_manager.cc:194] Registered new tserver with Master: 297774e99c8b4a8b8e5accac37cb6835 (127.7.90.66:33809)
I20260619 11:09:37.683957 7581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:53739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:37.698750 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:37.806413 7911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:37.806639 7911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:37.806679 7911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:37.810504 7911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:37.810545 7911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:37.810600 7911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:37.815773 7911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:42389
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:37.816630 7911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:37.817500 7911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:37.826610 7916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.826668 7919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:37.826676 7917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:37.826774 7911 server_base.cc:1034] running on GCE node
I20260619 11:09:37.827211 7911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:37.827831 7911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:37.829015 7911 hybrid_clock.cc:648] HybridClock initialized: now 1781867377829006 us; error 25 us; skew 500 ppm
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:37.831713 7911 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:37.832449 7911 webserver.cc:460] Webserver started at http://127.7.90.67:43755/ using document root <none> and password file <none>
I20260619 11:09:37.832921 7911 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:37.832976 7911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:37.833127 7911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:37.835033 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "bfc2319839304ecb8536ba0d48182c71"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "2a5b46fd28ec239296c042a321cbdd43"
server_key_iv: "7b6c110649f8b1f419f29557cc70e558"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.835597 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "bfc2319839304ecb8536ba0d48182c71"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "2a5b46fd28ec239296c042a321cbdd43"
server_key_iv: "7b6c110649f8b1f419f29557cc70e558"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.839042 7911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:37.841132 7926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.842327 7911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:37.842451 7911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "bfc2319839304ecb8536ba0d48182c71"
format_stamp: "Formatted at 2026-06-19 11:09:37 on dist-test-slave-f1t5"
server_key: "2a5b46fd28ec239296c042a321cbdd43"
server_key_iv: "7b6c110649f8b1f419f29557cc70e558"
server_key_version: "encryptionkey@0"
I20260619 11:09:37.842548 7911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:37.872922 7911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:37.876102 7911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:37.876293 7911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:37.876861 7911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:37.877671 7911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:37.877722 7911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.877790 7911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:37.877830 7911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:37.886309 7911 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:37695
I20260619 11:09:37.886339 8039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:37695 every 8 connection(s)
I20260619 11:09:37.887115 7911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:37.894452 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 7911
I20260619 11:09:37.894556 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:37.894831 7529 external_mini_cluster.cc:1428] Setting key 00716cd702c609b8bcea68890be1f769
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:37.899993 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.888739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:41637 (local address 127.7.90.126:42389)
0619 11:09:37.888876 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.888880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.889789 (+ 909us) server_negotiation.cc:407] Connection header received
0619 11:09:37.891075 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.891077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.891120 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.891192 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.892287 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.892791 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.893929 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.894134 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.894209 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.896896 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.896899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.896901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.896922 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.898338 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.898788 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.898790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.898792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.898859 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.899146 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.899148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.899149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.899301 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.899373 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.899786 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.899872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:09:37.900760 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.889138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:42389 (local address 127.7.90.67:41637)
0619 11:09:37.889632 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:37.889689 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:37.890862 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:37.891376 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:37.891389 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:37.891719 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:37.892172 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.892184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.892890 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.892892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.893822 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:37.893831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.894275 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.894279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:37.894495 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.895107 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:37.895127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:37.896744 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:37.898442 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.898447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.898457 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.898698 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.898947 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:37.898949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:37.898950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:37.899071 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:37.899381 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:37.899385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:37.899597 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 11:09:37.899742 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 11:09:37.899965 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260619 11:09:37.901823 8040 heartbeater.cc:347] Connected to a master server at 127.7.90.126:42389
I20260619 11:09:37.902019 8040 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:37.902420 8040 heartbeater.cc:511] Master 127.7.90.126:42389 requested a full tablet report, sending...
I20260619 11:09:37.903303 7581 ts_manager.cc:194] Registered new tserver with Master: bfc2319839304ecb8536ba0d48182c71 (127.7.90.67:37695)
I20260619 11:09:37.903848 7581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:41637
I20260619 11:09:37.908891 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:37.932811 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.924402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.7.90.65:37123)
0619 11:09:37.924683 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.924694 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.924723 (+ 29us) server_negotiation.cc:407] Connection header received
0619 11:09:37.924851 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.924856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.925026 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.925152 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.925829 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.926585 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.927318 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.927631 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.927725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.929912 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.929926 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.929938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.929959 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.931348 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.931690 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.931694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.931695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.931763 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.931979 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.931980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.931982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.932198 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.932337 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.932477 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.932624 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
W20260619 11:09:37.933738 7683 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44698
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:09:37 dist-test-slave-f1t5 krb5kdc[7536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:37.957015 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:37.949675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.7.90.65:37123)
0619 11:09:37.949802 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:37.949806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:37.949926 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:09:37.950013 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:37.950015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:37.950050 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:37.950110 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:37.950717 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.951207 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.951950 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:37.952156 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:37.952227 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:37.954352 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:37.954355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:37.954357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:37.954378 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:37.955737 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.956058 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.956061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.956062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.956127 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:37.956329 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:37.956331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:37.956332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:37.956468 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:37.956556 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:37.956721 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 11:09:37.956804 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:09:37.957998 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 7643
I20260619 11:09:37.963595 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 7777
I20260619 11:09:37.968852 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 7911
I20260619 11:09:37.973693 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 7549
2026-06-19T11:09:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2120 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:38 dist-test-slave-f1t5 krb5kdc[8054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:38 dist-test-slave-f1t5 krb5kdc[8054](info): set up 2 sockets
Jun 19 11:09:38 dist-test-slave-f1t5 krb5kdc[8054](info): commencing operation
krb5kdc: starting...
W20260619 11:09:39.001031 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:39.116671 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40199
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:39.224596 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:39.224916 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:39.224992 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:39.229079 8067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:39.229171 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:39.229213 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:39.229274 8067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:39.229317 8067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:39.234529 8067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:39.235615 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:39.236611 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:39.244848 8075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.244885 8073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:39.244983 8067 server_base.cc:1034] running on GCE node
W20260619 11:09:39.244901 8072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:39.245733 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:39.246749 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:39.247902 8067 hybrid_clock.cc:648] HybridClock initialized: now 1781867379247877 us; error 43 us; skew 500 ppm
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:39.250495 8067 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:39.251299 8067 webserver.cc:460] Webserver started at http://127.7.90.126:44235/ using document root <none> and password file <none>
I20260619 11:09:39.251801 8067 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:39.251911 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:39.252130 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:39.254110 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "d6188aba6fe543b18c0ace28d0d6ad3b"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "801536ff4b59519ddb04903b5dca6183"
server_key_iv: "c34c8b39dbbc6311910221f2b8bb2eda"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.254634 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "d6188aba6fe543b18c0ace28d0d6ad3b"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "801536ff4b59519ddb04903b5dca6183"
server_key_iv: "c34c8b39dbbc6311910221f2b8bb2eda"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.258097 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:39.260507 8082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.261719 8067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:39.261870 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "d6188aba6fe543b18c0ace28d0d6ad3b"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "801536ff4b59519ddb04903b5dca6183"
server_key_iv: "c34c8b39dbbc6311910221f2b8bb2eda"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.262001 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:39.275357 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:39.278515 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:39.278729 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:39.285424 8067 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40199
I20260619 11:09:39.285490 8134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40199 every 8 connection(s)
I20260619 11:09:39.286425 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:39.289065 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:39.292408 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8067
I20260619 11:09:39.292516 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:39.292762 7529 external_mini_cluster.cc:1428] Setting key aa3f1cd561737bb7f12eba1177e04ba9
I20260619 11:09:39.294776 8135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: Bootstrap starting.
I20260619 11:09:39.297353 8135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:39.298205 8135 log.cc:824] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:39.300307 8135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: No bootstrap required, opened a new log
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:39.304302 8135 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } }
I20260619 11:09:39.304589 8135 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:39.304636 8135 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6188aba6fe543b18c0ace28d0d6ad3b, State: Initialized, Role: FOLLOWER
I20260619 11:09:39.305130 8135 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } }
I20260619 11:09:39.305300 8135 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:39.305365 8135 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:39.305460 8135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:39.306573 8135 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } }
I20260619 11:09:39.306934 8135 leader_election.cc:302] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6188aba6fe543b18c0ace28d0d6ad3b; no voters:
I20260619 11:09:39.307297 8135 leader_election.cc:288] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:39.307425 8140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:39.307713 8140 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [term 1 LEADER]: Becoming Leader. State: Replica: d6188aba6fe543b18c0ace28d0d6ad3b, State: Running, Role: LEADER
I20260619 11:09:39.308104 8140 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } }
I20260619 11:09:39.308413 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.294150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.7.90.126:40199)
0619 11:09:39.294562 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:39.294580 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:39.294621 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:09:39.295421 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:39.295454 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:39.295856 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:39.296216 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:39.296827 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.298337 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.299211 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.299864 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.300024 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.302644 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:39.302650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:39.302666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:39.302694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:39.305129 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.305664 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.305673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.305682 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.305816 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.306081 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.306084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.306086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.306624 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:39.306811 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:39.307183 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 11:09:39.307494 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260619 11:09:39.308580 8135 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:39.309842 8142 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6188aba6fe543b18c0ace28d0d6ad3b. Latest consensus state: current_term: 1 leader_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } } }
I20260619 11:09:39.309963 8142 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:39.310350 8141 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6188aba6fe543b18c0ace28d0d6ad3b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40199 } } }
I20260619 11:09:39.310451 8141 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b [sys.catalog]: This master's current role is: LEADER
W20260619 11:09:39.313246 8155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:39.313313 8155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:39.313382 8150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:39.314083 8150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:39.319232 8150 catalog_manager.cc:1347] Generated new cluster ID: 876a7ddb26a442dd9dba4c36b15be756
I20260619 11:09:39.319295 8150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:39.330770 8150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:39.331619 8150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:39.338542 8150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6188aba6fe543b18c0ace28d0d6ad3b: Generated new TSK 0
I20260619 11:09:39.339250 8150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:39.372310 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:39.480911 8161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:39.481134 8161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:39.481173 8161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:39.485086 8161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:39.485139 8161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:39.485195 8161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:39.490187 8161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:39.491186 8161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:39.492174 8161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:39.501171 8166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.501158 8167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.501159 8169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:39.501358 8161 server_base.cc:1034] running on GCE node
I20260619 11:09:39.501832 8161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:39.502458 8161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:39.503636 8161 hybrid_clock.cc:648] HybridClock initialized: now 1781867379503620 us; error 31 us; skew 500 ppm
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:39.506218 8161 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:39.507030 8161 webserver.cc:460] Webserver started at http://127.7.90.65:36473/ using document root <none> and password file <none>
I20260619 11:09:39.507597 8161 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:39.507684 8161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:39.507922 8161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:39.510126 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "f8757bbbfb7a45f08b936ae18a464ccd"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "fc41e286608c72976f9c06d7c6662054"
server_key_iv: "e33f7eb04cb15a1ad79f85571b1e705c"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.510674 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "f8757bbbfb7a45f08b936ae18a464ccd"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "fc41e286608c72976f9c06d7c6662054"
server_key_iv: "e33f7eb04cb15a1ad79f85571b1e705c"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.514216 8161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:09:39.516456 8176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.517725 8161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:39.517844 8161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "f8757bbbfb7a45f08b936ae18a464ccd"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "fc41e286608c72976f9c06d7c6662054"
server_key_iv: "e33f7eb04cb15a1ad79f85571b1e705c"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.517949 8161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:39.535393 8161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:39.538156 8161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:39.538335 8161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:39.538846 8161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:39.539674 8161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:39.539742 8161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.539811 8161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:39.539853 8161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.548555 8161 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36949
I20260619 11:09:39.548591 8289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36949 every 8 connection(s)
I20260619 11:09:39.549353 8161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:39.558019 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8161
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:39.558106 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:39.558367 7529 external_mini_cluster.cc:1428] Setting key d66bc8ac4aa658bd45b62cfdec4c0a7e
I20260619 11:09:39.562145 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.550957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:47109 (local address 127.7.90.126:40199)
0619 11:09:39.551115 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:39.551120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:39.551880 (+ 760us) server_negotiation.cc:407] Connection header received
0619 11:09:39.553082 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:39.553084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:39.553129 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:39.553201 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:39.554396 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.554932 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.556044 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.556271 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.556381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.558667 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:39.558672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:39.558680 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:39.558701 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:39.560091 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.560593 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.560598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.560599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.560677 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.561061 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.561064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.561065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.561236 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:39.561316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:39.561936 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 11:09:39.562032 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:09:39.563361 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.551330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40199 (local address 127.7.90.65:47109)
0619 11:09:39.551738 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:39.551777 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:39.552906 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:39.553356 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:39.553362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:39.553787 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:39.554248 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.554261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.555054 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.555058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.555918 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.555927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.556422 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.556426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.556584 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.557081 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:39.557096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:39.558517 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:39.560222 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.560227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.560240 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.560491 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.560767 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.560772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.560774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.560949 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.561340 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:39.561347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:39.561686 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 11:09:39.561903 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 11:09:39.562316 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20260619 11:09:39.565131 8290 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40199
I20260619 11:09:39.565359 8290 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:39.565905 8290 heartbeater.cc:511] Master 127.7.90.126:40199 requested a full tablet report, sending...
I20260619 11:09:39.567327 8099 ts_manager.cc:194] Registered new tserver with Master: f8757bbbfb7a45f08b936ae18a464ccd (127.7.90.65:36949)
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:09:39.568439 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:47109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:39.584617 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:39.689994 8295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:39.690269 8295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:39.690344 8295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:39.694334 8295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:39.694404 8295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:39.694489 8295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:39.699592 8295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:39.700539 8295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:39.701501 8295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:39.710958 8301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.710889 8303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.710901 8300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:39.710908 8295 server_base.cc:1034] running on GCE node
I20260619 11:09:39.711576 8295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:39.712162 8295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:39.713340 8295 hybrid_clock.cc:648] HybridClock initialized: now 1781867379713327 us; error 27 us; skew 500 ppm
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:39.715878 8295 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:39.716622 8295 webserver.cc:460] Webserver started at http://127.7.90.66:34079/ using document root <none> and password file <none>
I20260619 11:09:39.717144 8295 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:39.717231 8295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:39.717446 8295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:39.719337 8295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "416b91ec93be4d9d8a522392e1ba9115"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "61c7b14e163c4e2837f67fa6ee937aaa"
server_key_iv: "676587d2143bdfe6d135d84afce17efa"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.719866 8295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "416b91ec93be4d9d8a522392e1ba9115"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "61c7b14e163c4e2837f67fa6ee937aaa"
server_key_iv: "676587d2143bdfe6d135d84afce17efa"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.723357 8295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:09:39.725586 8310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.726650 8295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:39.726778 8295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "416b91ec93be4d9d8a522392e1ba9115"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "61c7b14e163c4e2837f67fa6ee937aaa"
server_key_iv: "676587d2143bdfe6d135d84afce17efa"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.726881 8295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:39.737389 8295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:39.740020 8295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:39.740196 8295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:39.740734 8295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:39.741479 8295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:39.741567 8295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.741657 8295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:39.741701 8295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.749902 8295 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:39575
I20260619 11:09:39.749922 8423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:39575 every 8 connection(s)
I20260619 11:09:39.750721 8295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:39.760599 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8295
I20260619 11:09:39.760722 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:39.761032 7529 external_mini_cluster.cc:1428] Setting key 4bed9b643c1664021ddc558cc4b95080
I20260619 11:09:39.764101 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.752430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:54615 (local address 127.7.90.126:40199)
0619 11:09:39.752555 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:39.752558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:39.753302 (+ 744us) server_negotiation.cc:407] Connection header received
0619 11:09:39.754539 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:39.754542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:39.754585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:39.754656 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:39.755678 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.756182 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.757195 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.757400 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.757474 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.760439 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:39.760445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:39.760451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:39.760483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:39.761912 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.762449 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.762452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.762453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.762577 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.762899 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.762904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.762907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.763180 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:39.763288 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:39.763715 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 11:09:39.763851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:39.764765 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.752768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40199 (local address 127.7.90.66:54615)
0619 11:09:39.753161 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:39.753199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:39.754356 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:39.754806 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:39.754812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:39.755133 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:39.755559 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.755570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.756276 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.756278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.757100 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.757108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.757513 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.757535 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.757758 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.758385 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:39.758403 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:39.760271 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:39.762013 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.762017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.762030 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.762344 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.762668 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.762671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.762672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.762789 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.763280 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:39.763284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:39.763492 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 11:09:39.763637 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 11:09:39.763865 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260619 11:09:39.765892 8424 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40199
I20260619 11:09:39.766137 8424 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:39.766618 8424 heartbeater.cc:511] Master 127.7.90.126:40199 requested a full tablet report, sending...
I20260619 11:09:39.767587 8099 ts_manager.cc:194] Registered new tserver with Master: 416b91ec93be4d9d8a522392e1ba9115 (127.7.90.66:39575)
I20260619 11:09:39.768136 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:54615
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:39.787571 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:39.897724 8429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:39.897970 8429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:39.898012 8429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:39.901939 8429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:39.901986 8429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:39.902076 8429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:39.907141 8429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:39.908125 8429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:39.909054 8429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:39.917744 8437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.917788 8434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:39.917788 8435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:39.918073 8429 server_base.cc:1034] running on GCE node
I20260619 11:09:39.918530 8429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:39.919111 8429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:39.920284 8429 hybrid_clock.cc:648] HybridClock initialized: now 1781867379920252 us; error 44 us; skew 500 ppm
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:39.922928 8429 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:39.923697 8429 webserver.cc:460] Webserver started at http://127.7.90.67:46077/ using document root <none> and password file <none>
I20260619 11:09:39.924214 8429 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:39.924299 8429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:39.924515 8429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:39.926501 8429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "68a32fe789b0490bac2885a5108870e4"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "8c7da92eecfade93fa51f69cc1e7ee76"
server_key_iv: "6e77f3271ef2eb5a133fd8af805523b9"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.927063 8429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "68a32fe789b0490bac2885a5108870e4"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "8c7da92eecfade93fa51f69cc1e7ee76"
server_key_iv: "6e77f3271ef2eb5a133fd8af805523b9"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.930480 8429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:09:39.932570 8444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.933684 8429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:39.933802 8429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "68a32fe789b0490bac2885a5108870e4"
format_stamp: "Formatted at 2026-06-19 11:09:39 on dist-test-slave-f1t5"
server_key: "8c7da92eecfade93fa51f69cc1e7ee76"
server_key_iv: "6e77f3271ef2eb5a133fd8af805523b9"
server_key_version: "encryptionkey@0"
I20260619 11:09:39.933900 8429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:39.950687 8429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:39.953148 8429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:39.953328 8429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:39.953904 8429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:39.954631 8429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:39.954701 8429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.954769 8429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:39.954821 8429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:39.963168 8429 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:37089
I20260619 11:09:39.963191 8557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:37089 every 8 connection(s)
I20260619 11:09:39.963943 8429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:39.965934 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8429
I20260619 11:09:39.966068 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:39.966305 7529 external_mini_cluster.cc:1428] Setting key a6578304c6d0f4b9d07bdcb6ebcdc45c
Jun 19 11:09:39 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:39.976413 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.965580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:60619 (local address 127.7.90.126:40199)
0619 11:09:39.965751 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:39.965756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:39.966720 (+ 964us) server_negotiation.cc:407] Connection header received
0619 11:09:39.967948 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:39.967951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:39.968001 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:39.968081 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:39.969065 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.969583 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.970573 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.970764 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.970836 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.973077 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:39.973080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:39.973082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:39.973101 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:39.974301 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.974741 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.974746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.974748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.974833 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.975273 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.975278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.975279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:39.975548 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:39.975675 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:39.976211 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 11:09:39.976295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 11:09:39.977113 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.966078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40199 (local address 127.7.90.67:60619)
0619 11:09:39.966555 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:39.966604 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:39.967770 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:39.968220 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:39.968226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:39.968533 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:39.968954 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.968965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.969658 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.969661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.970480 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:39.970488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.970875 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.970878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:39.971011 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.971573 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:39.971588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:39.972941 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:39.974396 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.974400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.974409 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.974634 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.974958 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:39.974961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:39.974962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:39.975167 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:39.975665 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:39.975671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:39.976003 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 11:09:39.976151 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:09:39.976361 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260619 11:09:39.978080 8558 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40199
I20260619 11:09:39.978274 8558 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:39.978708 8558 heartbeater.cc:511] Master 127.7.90.126:40199 requested a full tablet report, sending...
I20260619 11:09:39.979713 8099 ts_manager.cc:194] Registered new tserver with Master: 68a32fe789b0490bac2885a5108870e4 (127.7.90.67:37089)
I20260619 11:09:39.980180 8099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:60619
I20260619 11:09:39.991020 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:09:40.001168 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:39.994233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.7.90.126:40199)
0619 11:09:39.994481 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:39.994485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:39.994624 (+ 139us) server_negotiation.cc:407] Connection header received
0619 11:09:39.994765 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:39.994768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:39.994811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:39.994892 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:39.995591 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.996099 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.996910 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:39.997123 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:39.997229 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:39.998436 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:39.998440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:39.998442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:39.998465 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:39.999567 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:39.999969 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:39.999972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:39.999974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.000047 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.000345 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.000349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.000351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.000544 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:40.000673 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.000897 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.001011 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260619 11:09:40.005316 8099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:40.008201 8099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:40.022356 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.016689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36949 (local address 127.0.0.1:47378)
0619 11:09:40.017335 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.017348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.017464 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.018308 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.018311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.018325 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.018413 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.018420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019495 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.019499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.020632 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.020642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021770 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.021774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.021966 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.022009 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.022112 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.022211 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":136,"thread_start_us":96,"threads_started":1}
I20260619 11:09:40.022361 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.017335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.7.90.66:39575)
0619 11:09:40.017731 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.017743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.017766 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:09:40.017840 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.017845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.018026 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.018170 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.018423 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019483 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.020890 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021631 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.021746 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.021895 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.022103 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.022201 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":99,"threads_started":1}
I20260619 11:09:40.022462 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.016970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:39575 (local address 127.0.0.1:58086)
0619 11:09:40.017649 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.017668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.017727 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.018170 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.018173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.018190 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.018307 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.018312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019610 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.019614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.020768 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.020778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021744 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.021748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.021848 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.021937 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.022057 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.022250 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":91,"thread_start_us":91,"threads_started":1}
I20260619 11:09:40.022821 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.017062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.7.90.65:36949)
0619 11:09:40.017902 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.017913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.017930 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:09:40.017987 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.017991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.018166 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.018534 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.018644 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019359 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.020751 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021628 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.021815 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.021957 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.022557 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.022658 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":101,"threads_started":1}
I20260619 11:09:40.023507 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.016379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.7.90.67:37089)
0619 11:09:40.016794 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.016806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.016830 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:09:40.016921 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.016926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.017132 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.017877 (+ 745us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.018034 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019213 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.020839 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021628 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.023059 (+ 1431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.023210 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.023248 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.023352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":92,"threads_started":1}
I20260619 11:09:40.022413 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.016262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:37089 (local address 127.0.0.1:56714)
0619 11:09:40.016567 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.016611 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.016725 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.017538 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.017546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.017574 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.017717 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.017725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.019348 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.019351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.020729 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.020740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.021770 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.021774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.021961 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.022035 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.022081 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.022250 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":182,"thread_start_us":83,"threads_started":1}
I20260619 11:09:40.025858 8224 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ee5df76904343a18ff5886470aaf2 (DEFAULT_TABLE table=test-table [id=f375f734118f424793acc1967eb75957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:40.026985 8224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1ee5df76904343a18ff5886470aaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:40.027388 8492 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ee5df76904343a18ff5886470aaf2 (DEFAULT_TABLE table=test-table [id=f375f734118f424793acc1967eb75957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:40.028276 8492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1ee5df76904343a18ff5886470aaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:40.030244 8358 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1ee5df76904343a18ff5886470aaf2 (DEFAULT_TABLE table=test-table [id=f375f734118f424793acc1967eb75957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:40.031097 8358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1ee5df76904343a18ff5886470aaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:40.034523 8575 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Bootstrap starting.
I20260619 11:09:40.035593 8576 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Bootstrap starting.
I20260619 11:09:40.036643 8577 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Bootstrap starting.
I20260619 11:09:40.037177 8575 tablet_bootstrap.cc:654] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:40.037463 8576 tablet_bootstrap.cc:654] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:40.038390 8575 log.cc:824] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:40.038690 8576 log.cc:824] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:40.039346 8577 tablet_bootstrap.cc:654] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:40.040092 8577 log.cc:824] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:40.040443 8575 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: No bootstrap required, opened a new log
I20260619 11:09:40.040627 8575 ts_tablet_manager.cc:1397] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:40.041100 8576 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: No bootstrap required, opened a new log
I20260619 11:09:40.041260 8576 ts_tablet_manager.cc:1397] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:09:40.041863 8577 tablet_bootstrap.cc:492] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: No bootstrap required, opened a new log
I20260619 11:09:40.042025 8577 ts_tablet_manager.cc:1397] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:40.043584 8575 raft_consensus.cc:348] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.043813 8575 raft_consensus.cc:374] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:40.043881 8575 raft_consensus.cc:729] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8757bbbfb7a45f08b936ae18a464ccd, State: Initialized, Role: FOLLOWER
I20260619 11:09:40.044309 8575 consensus_queue.cc:260] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.044948 8290 heartbeater.cc:503] Master 127.7.90.126:40199 was elected leader, sending a full tablet report...
I20260619 11:09:40.045277 8575 ts_tablet_manager.cc:1428] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:40.045608 8576 raft_consensus.cc:348] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.045861 8576 raft_consensus.cc:374] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:40.045918 8576 raft_consensus.cc:729] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a32fe789b0490bac2885a5108870e4, State: Initialized, Role: FOLLOWER
I20260619 11:09:40.046063 8577 raft_consensus.cc:348] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.046294 8577 raft_consensus.cc:374] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:40.046341 8577 raft_consensus.cc:729] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416b91ec93be4d9d8a522392e1ba9115, State: Initialized, Role: FOLLOWER
I20260619 11:09:40.046511 8576 consensus_queue.cc:260] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.046723 8577 consensus_queue.cc:260] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.047135 8558 heartbeater.cc:503] Master 127.7.90.126:40199 was elected leader, sending a full tablet report...
I20260619 11:09:40.047392 8576 ts_tablet_manager.cc:1428] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:09:40.047447 8577 ts_tablet_manager.cc:1428] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:09:40.047708 8424 heartbeater.cc:503] Master 127.7.90.126:40199 was elected leader, sending a full tablet report...
W20260619 11:09:40.050366 8291 tablet.cc:2369] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:09:40.214854 8559 tablet.cc:2369] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:09:40.251824 8425 tablet.cc:2369] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:40.361656 8583 raft_consensus.cc:482] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:40.361927 8583 raft_consensus.cc:504] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.363057 8583 leader_election.cc:288] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a32fe789b0490bac2885a5108870e4 (127.7.90.67:37089), f8757bbbfb7a45f08b936ae18a464ccd (127.7.90.65:36949)
I20260619 11:09:40.364503 8582 raft_consensus.cc:482] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:40.364696 8582 raft_consensus.cc:504] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.366243 8582 leader_election.cc:288] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416b91ec93be4d9d8a522392e1ba9115 (127.7.90.66:39575), f8757bbbfb7a45f08b936ae18a464ccd (127.7.90.65:36949)
I20260619 11:09:40.367808 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.363436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36949 (local address 127.7.90.66:36877)
0619 11:09:40.363734 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.363754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.363913 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.364200 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.364203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.364241 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.364396 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.364414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.365395 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.365399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.366613 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.366624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367407 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.367410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.367515 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.367537 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.367599 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.367661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":20,"spinlock_wait_cycles":61824,"thread_start_us":106,"threads_started":1}
I20260619 11:09:40.368180 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.363547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36877 (local address 127.7.90.65:36949)
0619 11:09:40.363764 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.363770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.363863 (+ 93us) server_negotiation.cc:407] Connection header received
0619 11:09:40.364041 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.364043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.364084 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.364180 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.364582 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.365269 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.366709 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367291 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.367395 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.367452 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.367996 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.368056 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260619 11:09:40.368250 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.363531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:54775 (local address 127.7.90.67:37089)
0619 11:09:40.363690 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.363695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.363808 (+ 113us) server_negotiation.cc:407] Connection header received
0619 11:09:40.364004 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.364010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.364061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.364120 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.364469 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.365158 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.366981 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367529 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.367593 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.367642 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.368085 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.368146 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:09:40.368750 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.363424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:37089 (local address 127.7.90.66:54775)
0619 11:09:40.363703 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.363718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.363885 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.364165 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.364171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.364187 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.364349 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.364354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.365272 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.365276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.366862 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.366869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367719 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.367722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.367809 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.367829 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.368553 (+ 724us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.368622 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":3,"spinlock_wait_cycles":23424,"thread_start_us":87,"threads_started":1}
I20260619 11:09:40.368853 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "416b91ec93be4d9d8a522392e1ba9115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" is_pre_election: true
I20260619 11:09:40.369148 8244 raft_consensus.cc:2454] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416b91ec93be4d9d8a522392e1ba9115 in term 0.
I20260619 11:09:40.369431 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "416b91ec93be4d9d8a522392e1ba9115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a32fe789b0490bac2885a5108870e4" is_pre_election: true
I20260619 11:09:40.369616 8512 raft_consensus.cc:2454] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416b91ec93be4d9d8a522392e1ba9115 in term 0.
I20260619 11:09:40.369624 8311 leader_election.cc:302] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416b91ec93be4d9d8a522392e1ba9115, f8757bbbfb7a45f08b936ae18a464ccd; no voters:
I20260619 11:09:40.369879 8583 raft_consensus.cc:2781] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:40.369973 8583 raft_consensus.cc:482] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:40.370031 8583 raft_consensus.cc:3037] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:40.370100 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.366004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:39575 (local address 127.7.90.67:35805)
0619 11:09:40.366193 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.366206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.366270 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.366761 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.366764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.366780 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.366879 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.366885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367719 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.367721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.368858 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.368868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.369749 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.369752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.369840 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.369859 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.369920 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.369978 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":47}
I20260619 11:09:40.370386 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.366407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:35805 (local address 127.7.90.66:39575)
0619 11:09:40.366493 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.366497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.366510 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:40.366546 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.366549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.366587 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.366661 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.366998 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.367619 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.368979 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.369571 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.369655 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.369700 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.370222 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.370289 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 11:09:40.370635 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.366039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36949 (local address 127.7.90.67:43633)
0619 11:09:40.366458 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:40.366471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:40.366551 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:40.367127 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:40.367129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:40.367144 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:40.367233 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.367238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.368176 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.368179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.369319 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:40.369326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.370261 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.370264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:40.370348 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.370368 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:09:40.370438 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:09:40.370499 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260619 11:09:40.371011 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.366578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:43633 (local address 127.7.90.65:36949)
0619 11:09:40.366872 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.366876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.366887 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:09:40.366920 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.366922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.366957 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.367026 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:40.367350 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.368072 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.369434 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.370052 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.370140 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.370195 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.370780 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.370857 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":223,"thread_start_us":64,"threads_started":1}
I20260619 11:09:40.371261 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "68a32fe789b0490bac2885a5108870e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416b91ec93be4d9d8a522392e1ba9115" is_pre_election: true
I20260619 11:09:40.371337 8583 raft_consensus.cc:504] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.371388 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "68a32fe789b0490bac2885a5108870e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" is_pre_election: true
I20260619 11:09:40.371501 8378 raft_consensus.cc:2379] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68a32fe789b0490bac2885a5108870e4 in current term 1: Already voted for candidate 416b91ec93be4d9d8a522392e1ba9115 in this term.
I20260619 11:09:40.371515 8244 raft_consensus.cc:2454] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a32fe789b0490bac2885a5108870e4 in term 0.
I20260619 11:09:40.371611 8583 leader_election.cc:288] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [CANDIDATE]: Term 1 election: Requested vote from peers 68a32fe789b0490bac2885a5108870e4 (127.7.90.67:37089), f8757bbbfb7a45f08b936ae18a464ccd (127.7.90.65:36949)
I20260619 11:09:40.372006 8445 leader_election.cc:302] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a32fe789b0490bac2885a5108870e4, f8757bbbfb7a45f08b936ae18a464ccd; no voters:
I20260619 11:09:40.372013 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "416b91ec93be4d9d8a522392e1ba9115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a32fe789b0490bac2885a5108870e4"
I20260619 11:09:40.372167 8512 raft_consensus.cc:3037] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:40.372588 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1ee5df76904343a18ff5886470aaf2" candidate_uuid: "416b91ec93be4d9d8a522392e1ba9115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8757bbbfb7a45f08b936ae18a464ccd"
I20260619 11:09:40.372710 8244 raft_consensus.cc:3037] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:40.373176 8512 raft_consensus.cc:2454] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416b91ec93be4d9d8a522392e1ba9115 in term 1.
I20260619 11:09:40.373276 8582 raft_consensus.cc:2741] T 6a1ee5df76904343a18ff5886470aaf2 P 68a32fe789b0490bac2885a5108870e4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 11:09:40.373483 8314 leader_election.cc:302] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416b91ec93be4d9d8a522392e1ba9115, 68a32fe789b0490bac2885a5108870e4; no voters:
I20260619 11:09:40.373725 8583 raft_consensus.cc:2781] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:40.373921 8583 raft_consensus.cc:686] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [term 1 LEADER]: Becoming Leader. State: Replica: 416b91ec93be4d9d8a522392e1ba9115, State: Running, Role: LEADER
I20260619 11:09:40.373935 8244 raft_consensus.cc:2454] T 6a1ee5df76904343a18ff5886470aaf2 P f8757bbbfb7a45f08b936ae18a464ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416b91ec93be4d9d8a522392e1ba9115 in term 1.
I20260619 11:09:40.374238 8583 consensus_queue.cc:237] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } }
I20260619 11:09:40.377188 8098 catalog_manager.cc:5495] T 6a1ee5df76904343a18ff5886470aaf2 P 416b91ec93be4d9d8a522392e1ba9115 reported cstate change: term changed from 0 to 1, leader changed from <none> to 416b91ec93be4d9d8a522392e1ba9115 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "416b91ec93be4d9d8a522392e1ba9115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a32fe789b0490bac2885a5108870e4" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416b91ec93be4d9d8a522392e1ba9115" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8757bbbfb7a45f08b936ae18a464ccd" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36949 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:40.473779 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.465253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.7.90.65:36949)
0619 11:09:40.465385 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.465389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.465595 (+ 206us) server_negotiation.cc:407] Connection header received
0619 11:09:40.465800 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.465803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.465851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.465921 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:40.466641 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.467135 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.467910 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.468121 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.468191 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.470523 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:40.470534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:40.470550 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:40.470571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:40.472271 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.472630 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.472635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.472636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.472717 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.472960 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.472962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.472963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.473181 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:40.473336 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.473475 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.473658 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:40.499428 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.491099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.7.90.65:36949)
0619 11:09:40.491225 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.491230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.491356 (+ 126us) server_negotiation.cc:407] Connection header received
0619 11:09:40.491496 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.491499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.491543 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.491640 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:40.492274 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.492765 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.493636 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.493864 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.493972 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.496168 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:40.496178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:40.496181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:40.496202 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:40.497305 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.497782 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.497786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.497787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.498255 (+ 468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.498530 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.498538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.498539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.498702 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:40.498833 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.499149 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.499270 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:40.523972 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:40.515501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.7.90.65:36949)
0619 11:09:40.515680 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:40.515685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:40.515869 (+ 184us) server_negotiation.cc:407] Connection header received
0619 11:09:40.516040 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:40.516043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:40.516096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:40.516188 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:40.516871 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.517387 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.518277 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:40.518499 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:40.518616 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:40.521037 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:40.521040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:40.521042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:40.521063 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:40.522445 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.522848 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.522853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.522855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.522931 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:40.523198 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:40.523201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:40.523202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:40.523357 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:40.523464 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:40.523658 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 11:09:40.523759 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 11:09:40.525364 8204 tablet_service.cc:2908] Rejecting scan request for tablet 6a1ee5df76904343a18ff5886470aaf2: Uninitialized: clean time has not yet been initialized
I20260619 11:09:40.527186 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8161
I20260619 11:09:40.532912 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8295
I20260619 11:09:40.538735 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8429
I20260619 11:09:40.543885 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8067
2026-06-19T11:09:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2570 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8602](info): set up 2 sockets
Jun 19 11:09:40 dist-test-slave-f1t5 krb5kdc[8602](info): commencing operation
krb5kdc: starting...
W20260619 11:09:41.569293 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:41 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:41.683663 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36863
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:41.788396 8615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:41.788631 8615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:41.788669 8615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:41.792668 8615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:41.792717 8615 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:09:41.792734 8615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:41.792752 8615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:41.792791 8615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:41.792827 8615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:41.797857 8615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:36863
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:41.798806 8615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:41.799759 8615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:41.807825 8621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:41.807883 8620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:41.807847 8623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:41.808094 8615 server_base.cc:1034] running on GCE node
I20260619 11:09:41.808601 8615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:41.809722 8615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:41.810874 8615 hybrid_clock.cc:648] HybridClock initialized: now 1781867381810846 us; error 41 us; skew 500 ppm
Jun 19 11:09:41 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:41.816388 8615 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:41.817416 8615 webserver.cc:460] Webserver started at http://127.7.90.126:37779/ using document root <none> and password file <none>
I20260619 11:09:41.818004 8615 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:41.818097 8615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:41.818320 8615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:41.820261 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "40f55735d960431080b337f526598ccd"
format_stamp: "Formatted at 2026-06-19 11:09:41 on dist-test-slave-f1t5"
server_key: "412a1a1fda3143af4f52ab04f412e219"
server_key_iv: "70421dab6425a14f2ffeb4e716521ad2"
server_key_version: "encryptionkey@0"
I20260619 11:09:41.820793 8615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "40f55735d960431080b337f526598ccd"
format_stamp: "Formatted at 2026-06-19 11:09:41 on dist-test-slave-f1t5"
server_key: "412a1a1fda3143af4f52ab04f412e219"
server_key_iv: "70421dab6425a14f2ffeb4e716521ad2"
server_key_version: "encryptionkey@0"
I20260619 11:09:41.824414 8615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:41.826668 8630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:41.827803 8615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:41.827945 8615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "40f55735d960431080b337f526598ccd"
format_stamp: "Formatted at 2026-06-19 11:09:41 on dist-test-slave-f1t5"
server_key: "412a1a1fda3143af4f52ab04f412e219"
server_key_iv: "70421dab6425a14f2ffeb4e716521ad2"
server_key_version: "encryptionkey@0"
I20260619 11:09:41.828049 8615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:41.842165 8615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:41.845376 8615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:41.845600 8615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:41.854959 8615 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:36863
I20260619 11:09:41.854979 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:36863 every 8 connection(s)
I20260619 11:09:41.855894 8615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:41.858520 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:41.859309 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8615
I20260619 11:09:41.859418 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:41.859694 7529 external_mini_cluster.cc:1428] Setting key 6b003035f01b69856578812ede38c833
I20260619 11:09:41.864308 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Bootstrap starting.
I20260619 11:09:41.866575 8693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:41.867257 8693 log.cc:824] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:41 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:41.869174 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: No bootstrap required, opened a new log
I20260619 11:09:41.872187 8693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } }
I20260619 11:09:41.872407 8693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:41.872521 8693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f55735d960431080b337f526598ccd, State: Initialized, Role: FOLLOWER
I20260619 11:09:41.872977 8693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } }
I20260619 11:09:41.873154 8693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:41.873250 8693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:41.873374 8693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:41.874405 8693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } }
I20260619 11:09:41.874747 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:41.861227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.7.90.126:36863)
0619 11:09:41.861700 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:41.861717 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:41.861752 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:09:41.862399 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:41.862416 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:41.862731 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:41.863076 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:41.863940 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:41.864986 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:41.865916 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:41.866385 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:41.866560 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:41.869342 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:41.869348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:41.869360 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:41.869387 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:41.871634 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:41.872090 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:41.872099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:41.872103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:41.872203 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:41.872494 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:41.872498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:41.872501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:41.872983 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:41.873209 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:41.873496 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 11:09:41.873812 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":129,"threads_started":1}
I20260619 11:09:41.874743 8693 leader_election.cc:302] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40f55735d960431080b337f526598ccd; no voters:
I20260619 11:09:41.875046 8693 leader_election.cc:288] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:41.875144 8698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:41.875360 8698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [term 1 LEADER]: Becoming Leader. State: Replica: 40f55735d960431080b337f526598ccd, State: Running, Role: LEADER
I20260619 11:09:41.875684 8698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } }
I20260619 11:09:41.876163 8693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:41.877295 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40f55735d960431080b337f526598ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } } }
I20260619 11:09:41.877460 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:41.877875 8707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:41.878830 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40f55735d960431080b337f526598ccd. Latest consensus state: current_term: 1 leader_uuid: "40f55735d960431080b337f526598ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40f55735d960431080b337f526598ccd" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36863 } } }
I20260619 11:09:41.878922 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:41.880724 8707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:41.885569 8707 catalog_manager.cc:1347] Generated new cluster ID: 99d4ab3d2e274f438ac691f75a6d1e78
I20260619 11:09:41.885661 8707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:41.899333 8707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:41.900486 8707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:41.914162 8707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Generated new TSK 0
I20260619 11:09:41.914853 8707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:41.940145 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:36863
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:42.045933 8719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:42.046156 8719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:42.046227 8719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:42.049998 8719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:42.050043 8719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:42.050109 8719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:42.050172 8719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:42.055198 8719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:42.056162 8719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:42.057126 8719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:42.066749 8724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.066751 8725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.066785 8727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:42.066830 8719 server_base.cc:1034] running on GCE node
I20260619 11:09:42.067348 8719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:42.067873 8719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:42.069026 8719 hybrid_clock.cc:648] HybridClock initialized: now 1781867382068999 us; error 41 us; skew 500 ppm
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:42.071563 8719 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:42.072338 8719 webserver.cc:460] Webserver started at http://127.7.90.65:43763/ using document root <none> and password file <none>
I20260619 11:09:42.072827 8719 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:42.072880 8719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:42.073036 8719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:42.074970 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "b2a4d596f4ec4959a19b4ceba8863745"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "cb65fc5a0644b6870c1ce067915f9454"
server_key_iv: "eeac3734603b1d230be6d9083227a8b0"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.075443 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "b2a4d596f4ec4959a19b4ceba8863745"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "cb65fc5a0644b6870c1ce067915f9454"
server_key_iv: "eeac3734603b1d230be6d9083227a8b0"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.079144 8719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:42.081238 8734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.082423 8719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:42.082538 8719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "b2a4d596f4ec4959a19b4ceba8863745"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "cb65fc5a0644b6870c1ce067915f9454"
server_key_iv: "eeac3734603b1d230be6d9083227a8b0"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.082631 8719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:42.102586 8719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:42.105108 8719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:42.105285 8719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:42.106635 8719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:42.106681 8719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.106719 8719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:42.106775 8719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:42.118017 8719 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:35259
I20260619 11:09:42.118094 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:35259 every 8 connection(s)
I20260619 11:09:42.118858 8719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:42.120239 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.107485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:36735 (local address 127.7.90.126:36863)
0619 11:09:42.107602 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.107606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.108545 (+ 939us) server_negotiation.cc:407] Connection header received
0619 11:09:42.109643 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.109647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.109709 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.109798 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.111052 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.111679 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.112832 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.113070 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.113182 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.116664 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.116669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.116677 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.116699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.118224 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.118718 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.118723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.118725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.118803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.119151 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.119156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.119158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.119383 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.119493 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.120020 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.120125 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:09:42.120991 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.107953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.65:36735)
0619 11:09:42.108374 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.108429 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.109355 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.110124 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.110129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.110416 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.110878 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.110897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.111814 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.111818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.112701 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.112709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.113213 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.113217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.113382 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.114020 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.114038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.116506 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.118340 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.118347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.118378 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.118617 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.118897 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.118900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.118902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.119060 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.119552 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.119562 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.119810 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.119961 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.120175 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260619 11:09:42.122257 8850 heartbeater.cc:347] Connected to a master server at 127.7.90.126:36863
I20260619 11:09:42.122406 8850 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:42.122848 8850 heartbeater.cc:511] Master 127.7.90.126:36863 requested a full tablet report, sending...
I20260619 11:09:42.124233 8646 ts_manager.cc:194] Registered new tserver with Master: b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259)
I20260619 11:09:42.125692 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8719
I20260619 11:09:42.125716 8646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:36735
I20260619 11:09:42.125804 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:42.126120 7529 external_mini_cluster.cc:1428] Setting key e14fd6702c6e9cad2636ca4dbb75be7e
I20260619 11:09:42.131040 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.124227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:32779 (local address 127.7.90.126:36863)
0619 11:09:42.124362 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.124366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.124545 (+ 179us) server_negotiation.cc:407] Connection header received
0619 11:09:42.124643 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.124645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.124679 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.124739 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.125195 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.125849 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.126839 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.127090 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.127207 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.128211 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.128214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.128217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.128238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.129569 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.129948 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.129952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.129957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.130037 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.130297 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.130301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.130303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.130465 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.130564 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.130796 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.130893 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":50}
I20260619 11:09:42.131055 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.124168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.65:32779)
0619 11:09:42.124460 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.124474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.124566 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.124776 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.124779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.124979 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.125084 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.125091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.125985 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.125988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.126722 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.126735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.127207 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.127210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.127295 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.127756 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.127767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.128050 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.129672 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.129674 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.129676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.129856 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.130123 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.130127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.130129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.130213 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.130555 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.130558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.130642 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.130749 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.130872 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:42.150506 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:36863
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:42.259289 8859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:42.259513 8859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:42.259591 8859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:42.263576 8859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:42.263624 8859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:42.263690 8859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:42.263770 8859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:42.268882 8859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:42.269892 8859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:42.270885 8859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:42.279992 8867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.280002 8865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.280071 8864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:42.280174 8859 server_base.cc:1034] running on GCE node
I20260619 11:09:42.280748 8859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:42.281280 8859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:42.282461 8859 hybrid_clock.cc:648] HybridClock initialized: now 1781867382282441 us; error 40 us; skew 500 ppm
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:42.285110 8859 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:42.286047 8859 webserver.cc:460] Webserver started at http://127.7.90.66:43081/ using document root <none> and password file <none>
I20260619 11:09:42.286625 8859 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:42.286713 8859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:42.286911 8859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:42.288877 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "2778fc6163dfe7a2fc948fede3eb0147"
server_key_iv: "8ed8bd517be2d64c17c1d87ea552af9a"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.289405 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "2778fc6163dfe7a2fc948fede3eb0147"
server_key_iv: "8ed8bd517be2d64c17c1d87ea552af9a"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.292968 8859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:09:42.295243 8874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.296329 8859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:42.296458 8859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "2778fc6163dfe7a2fc948fede3eb0147"
server_key_iv: "8ed8bd517be2d64c17c1d87ea552af9a"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.296557 8859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:42.314333 8859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:42.318881 8859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:42.319072 8859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:42.320437 8859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:42.320555 8859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.320652 8859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:42.320686 8859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:42.332007 8859 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:40173
I20260619 11:09:42.332084 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:40173 every 8 connection(s)
I20260619 11:09:42.333184 8859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:42.335813 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8859
I20260619 11:09:42.335953 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:42.336280 7529 external_mini_cluster.cc:1428] Setting key 0d52d64b49f5cd88d6bea5c7c9c12b6d
I20260619 11:09:42.337471 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.321859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.66:49903)
0619 11:09:42.322460 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.322544 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.323703 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.324606 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.324616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.325078 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.325869 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.325900 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.327058 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.327063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.328617 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.328629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.329255 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.329259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.329444 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.330227 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.330249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.332229 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.334364 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.334372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.334389 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.334730 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.335059 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.335064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.335068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.335251 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.335712 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.335720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.336012 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.336192 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.336497 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260619 11:09:42.337916 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.321340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:49903 (local address 127.7.90.126:36863)
0619 11:09:42.321537 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.321543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.322641 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 11:09:42.323999 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.324006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.324075 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.324141 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.326051 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.326930 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.328760 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.329075 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.329214 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.332471 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.332475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.332478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.332505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.334248 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.334831 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.334835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.334837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.334953 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.335358 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.335361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.335363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.335611 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.335702 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.337656 (+ 1954us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.337769 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:09:42.342826 8990 heartbeater.cc:347] Connected to a master server at 127.7.90.126:36863
I20260619 11:09:42.343109 8990 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:42.343756 8990 heartbeater.cc:511] Master 127.7.90.126:36863 requested a full tablet report, sending...
I20260619 11:09:42.344890 8646 ts_manager.cc:194] Registered new tserver with Master: 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173)
I20260619 11:09:42.345678 8646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:49903
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:09:42.351683 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.343173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.66:46307)
0619 11:09:42.343448 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.343463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.343559 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.343828 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.343831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.344092 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.344239 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.344252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.345190 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.345194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.346179 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.346190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.346725 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.346729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.346836 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.347378 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.347399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.347797 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.349641 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.349645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.349648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.349832 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.350176 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.350181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.350183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.350354 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.350945 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.350949 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.351071 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.351230 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.351399 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260619 11:09:42.351688 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.343031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:46307 (local address 127.7.90.126:36863)
0619 11:09:42.343150 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.343154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.343526 (+ 372us) server_negotiation.cc:407] Connection header received
0619 11:09:42.343639 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.343642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.343709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.343787 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.344372 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.345053 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.346293 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.346601 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.346707 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.347974 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.347978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.347980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.348006 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.349511 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.349944 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.349950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.349952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.350057 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.350474 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.350479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.350481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.350770 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.350853 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.351293 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.351422 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:42.367851 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:36863
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:42.481777 8999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:42.482025 8999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:42.482110 8999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:42.486159 8999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:42.486258 8999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:42.486320 8999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:42.486361 8999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:42.491487 8999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:36863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:42.492435 8999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:42.493355 8999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:42.502787 9005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.502833 9007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:42.502787 9004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:42.502995 8999 server_base.cc:1034] running on GCE node
I20260619 11:09:42.503508 8999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:42.504102 8999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:42.505280 8999 hybrid_clock.cc:648] HybridClock initialized: now 1781867382505259 us; error 30 us; skew 500 ppm
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:42.508021 8999 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:42.508975 8999 webserver.cc:460] Webserver started at http://127.7.90.67:36539/ using document root <none> and password file <none>
I20260619 11:09:42.509582 8999 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:42.509672 8999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:42.509908 8999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:42.511819 8999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "4093beb136354f068268a1ee70f64623"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "b66e6a20e999e5a550f05f0460110569"
server_key_iv: "7b577ff6f316d925e38585825c2f315e"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.512418 8999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "4093beb136354f068268a1ee70f64623"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "b66e6a20e999e5a550f05f0460110569"
server_key_iv: "7b577ff6f316d925e38585825c2f315e"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.516031 8999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:42.518440 9014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.519486 8999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:42.519644 8999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "4093beb136354f068268a1ee70f64623"
format_stamp: "Formatted at 2026-06-19 11:09:42 on dist-test-slave-f1t5"
server_key: "b66e6a20e999e5a550f05f0460110569"
server_key_iv: "7b577ff6f316d925e38585825c2f315e"
server_key_version: "encryptionkey@0"
I20260619 11:09:42.519752 8999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:42.539527 8999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:42.542580 8999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:42.542797 8999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:42.544332 8999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:42.544407 8999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:42.544480 8999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:42.544523 8999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:42.556259 8999 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:38425
I20260619 11:09:42.556303 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:38425 every 8 connection(s)
I20260619 11:09:42.557399 8999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:42.558995 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.545022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:50625 (local address 127.7.90.126:36863)
0619 11:09:42.545185 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.545190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.546627 (+ 1437us) server_negotiation.cc:407] Connection header received
0619 11:09:42.547822 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.547825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.547889 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.547996 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.549379 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.550106 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.551244 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.551469 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.551597 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.555024 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.555029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.555031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.555061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.556770 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.557250 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.557255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.557257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.557374 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.557806 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.557811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.557813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.558078 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.558195 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.558768 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.558866 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:09:42.559768 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.545999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.67:50625)
0619 11:09:42.546452 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.546504 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.547587 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.548181 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.548188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.548540 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.549223 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.549238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.550232 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.550235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.551120 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.551129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.551644 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.551648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.551850 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.552548 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.552567 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.554829 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.556885 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.556890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.556900 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.557145 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.557511 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.557515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.557518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.557697 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.558254 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.558262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.558534 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.558695 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.558941 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":131,"threads_started":1}
I20260619 11:09:42.561043 9130 heartbeater.cc:347] Connected to a master server at 127.7.90.126:36863
I20260619 11:09:42.561220 9130 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:42.561744 9130 heartbeater.cc:511] Master 127.7.90.126:36863 requested a full tablet report, sending...
I20260619 11:09:42.562711 8645 ts_manager.cc:194] Registered new tserver with Master: 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425)
I20260619 11:09:42.563474 8645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:50625
I20260619 11:09:42.564184 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 8999
I20260619 11:09:42.564293 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:42.564590 7529 external_mini_cluster.cc:1428] Setting key 9c44400ac3b3cf8f7ada752e4a3b2f43
I20260619 11:09:42.566609 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:09:42.570683 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.563113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:46507 (local address 127.7.90.126:36863)
0619 11:09:42.563712 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.563715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.563725 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:09:42.563749 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.563751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.563777 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.563891 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.564438 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.565159 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.566269 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.566580 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.566683 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.567696 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.567703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.567709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.567739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.569139 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.569633 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.569635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.569636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.569704 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.569985 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.569987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.569988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.570135 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.570206 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.570500 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.570580 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260619 11:09:42.570886 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.562991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.7.90.67:46507)
0619 11:09:42.563273 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.563287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.563361 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.563905 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.563908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.564124 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.564278 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.564286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.565277 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.565282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.566165 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.566174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.566714 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.566719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.566814 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.567277 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.567287 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.567562 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.569238 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.569241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.569246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.569508 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.569787 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.569793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.569798 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.569894 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.570222 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.570228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.570334 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.570446 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.570604 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":10,"thread_start_us":94,"threads_started":1}
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:42.591744 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.584390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.7.90.126:36863)
0619 11:09:42.584516 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.584519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.584531 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:42.584568 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.584570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.584605 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.584670 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.585302 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.585902 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.586713 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.586931 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.587045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.589195 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.589198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.589200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.589221 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.590471 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.590828 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.590830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.590832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.590897 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.591119 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.591121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.591122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.591272 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.591368 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.591548 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.591630 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:09:42.594485 8645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:42.597308 8645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:42.611796 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.605981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.7.90.65:35259)
0619 11:09:42.606275 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.606291 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.607039 (+ 748us) server_negotiation.cc:407] Connection header received
0619 11:09:42.607125 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.607130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.607284 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.607446 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.607682 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.608812 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.610450 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611130 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611255 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611435 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.611551 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.611643 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260619 11:09:42.611799 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.605373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35259 (local address 127.0.0.1:43010)
0619 11:09:42.606564 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.606578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.606675 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.607418 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.607421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.607439 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.607558 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.607565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.608926 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.608929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.610309 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.610320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611252 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.611369 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611410 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.611523 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.611644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":100,"thread_start_us":98,"threads_started":1}
I20260619 11:09:42.612128 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.605347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.7.90.66:40173)
0619 11:09:42.606882 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.606896 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.606914 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:09:42.606973 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.606976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.607128 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.607286 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.607583 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.608559 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.610456 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611178 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.611863 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.611969 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":127,"threads_started":1}
I20260619 11:09:42.612237 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.606569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:38425 (local address 127.0.0.1:47660)
0619 11:09:42.607539 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.607553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.607638 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.608056 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.608059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.608077 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.608174 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.608181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.609760 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.609767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.610645 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.610661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611863 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.611958 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611979 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.612048 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.612107 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260619 11:09:42.612423 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.605247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:40173 (local address 127.0.0.1:55992)
0619 11:09:42.605750 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.605794 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.605913 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.607274 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.607280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.607300 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.607449 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.607459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.608684 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.608690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.610309 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.610320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611488 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.611593 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611631 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.611693 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.611763 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":75,"thread_start_us":89,"threads_started":1}
I20260619 11:09:42.612690 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.606751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.7.90.67:38425)
0619 11:09:42.607051 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.607062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.607626 (+ 564us) server_negotiation.cc:407] Connection header received
0619 11:09:42.607741 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.607747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.607916 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.608056 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.608288 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.609657 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.610782 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.611554 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.611654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.611805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.612431 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.612549 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260619 11:09:42.615079 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 4cad846289304ca38e714c49f07a0125 (DEFAULT_TABLE table=test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.615198 9064 tablet_service.cc:1467] Processing CreateTablet for tablet 4cad846289304ca38e714c49f07a0125 (DEFAULT_TABLE table=test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.615198 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 4cad846289304ca38e714c49f07a0125 (DEFAULT_TABLE table=test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.616111 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cad846289304ca38e714c49f07a0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.616161 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cad846289304ca38e714c49f07a0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.616171 9064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cad846289304ca38e714c49f07a0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.622866 9153 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Bootstrap starting.
I20260619 11:09:42.624205 9154 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Bootstrap starting.
I20260619 11:09:42.625583 9153 tablet_bootstrap.cc:654] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.625819 9155 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Bootstrap starting.
I20260619 11:09:42.626313 9154 tablet_bootstrap.cc:654] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.626399 9153 log.cc:824] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:42.627110 9154 log.cc:824] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:42.627750 9155 tablet_bootstrap.cc:654] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.628276 9153 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: No bootstrap required, opened a new log
I20260619 11:09:42.628477 9153 ts_tablet_manager.cc:1397] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 11:09:42.628530 9155 log.cc:824] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:42.629062 9154 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: No bootstrap required, opened a new log
I20260619 11:09:42.629256 9154 ts_tablet_manager.cc:1397] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:09:42.630391 9155 tablet_bootstrap.cc:492] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: No bootstrap required, opened a new log
I20260619 11:09:42.630597 9155 ts_tablet_manager.cc:1397] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:42.631917 9153 raft_consensus.cc:348] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.632148 9153 raft_consensus.cc:374] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.632222 9153 raft_consensus.cc:729] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa7d6a494534780b6b7dbf3c33a3e9e, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.632676 9153 consensus_queue.cc:260] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.633152 9154 raft_consensus.cc:348] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.633349 8990 heartbeater.cc:503] Master 127.7.90.126:36863 was elected leader, sending a full tablet report...
I20260619 11:09:42.633479 9154 raft_consensus.cc:374] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.633617 9154 raft_consensus.cc:729] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a4d596f4ec4959a19b4ceba8863745, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.634093 9154 consensus_queue.cc:260] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.634645 9153 ts_tablet_manager.cc:1428] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260619 11:09:42.634514 9155 raft_consensus.cc:348] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.634724 9155 raft_consensus.cc:374] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.634827 9155 raft_consensus.cc:729] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4093beb136354f068268a1ee70f64623, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.635023 9154 ts_tablet_manager.cc:1428] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:42.635278 9155 consensus_queue.cc:260] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.635417 8850 heartbeater.cc:503] Master 127.7.90.126:36863 was elected leader, sending a full tablet report...
I20260619 11:09:42.635891 9130 heartbeater.cc:503] Master 127.7.90.126:36863 was elected leader, sending a full tablet report...
I20260619 11:09:42.636214 9155 ts_tablet_manager.cc:1428] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:42.658540 9161 raft_consensus.cc:482] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:42.658741 9161 raft_consensus.cc:504] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.659869 9161 leader_election.cc:288] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173), b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259)
I20260619 11:09:42.663782 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.660182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:50381 (local address 127.7.90.66:40173)
0619 11:09:42.660287 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.660293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.660405 (+ 112us) server_negotiation.cc:407] Connection header received
0619 11:09:42.660465 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.660467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.660508 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.660604 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.660970 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.661485 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.662672 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.663252 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.663356 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.663436 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.663622 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.663687 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:42.663787 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.660082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:40173 (local address 127.7.90.67:50381)
0619 11:09:42.660276 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.660291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.660387 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.660627 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.660633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.660651 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.660836 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.660864 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.661613 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.661617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.662547 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.662556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.663380 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.663384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.663490 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.663515 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.663591 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.663659 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 11:09:42.664423 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.660248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:40293 (local address 127.7.90.65:35259)
0619 11:09:42.660363 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.660369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.660589 (+ 220us) server_negotiation.cc:407] Connection header received
0619 11:09:42.660700 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.660704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.660763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.660821 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.661126 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.661711 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.663359 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.663937 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.664041 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.664111 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.664283 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.664340 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:09:42.664445 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.660115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35259 (local address 127.7.90.67:40293)
0619 11:09:42.660477 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.660492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.660601 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.660850 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.660853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.660868 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.661019 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.661024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.661827 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.661832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.663227 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.663237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.664040 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.664044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.664143 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.664165 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.664264 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.664325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
I20260619 11:09:42.664537 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cad846289304ca38e714c49f07a0125" candidate_uuid: "4093beb136354f068268a1ee70f64623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" is_pre_election: true
I20260619 11:09:42.664860 8944 raft_consensus.cc:2454] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4093beb136354f068268a1ee70f64623 in term 0.
I20260619 11:09:42.665174 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cad846289304ca38e714c49f07a0125" candidate_uuid: "4093beb136354f068268a1ee70f64623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a4d596f4ec4959a19b4ceba8863745" is_pre_election: true
I20260619 11:09:42.665329 9018 leader_election.cc:302] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7d6a494534780b6b7dbf3c33a3e9e, 4093beb136354f068268a1ee70f64623; no voters:
I20260619 11:09:42.665462 8804 raft_consensus.cc:2454] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4093beb136354f068268a1ee70f64623 in term 0.
I20260619 11:09:42.665637 9161 raft_consensus.cc:2781] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:42.665719 9161 raft_consensus.cc:482] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:42.665793 9161 raft_consensus.cc:3037] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.666787 9161 raft_consensus.cc:504] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.667169 9161 leader_election.cc:288] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [CANDIDATE]: Term 1 election: Requested vote from peers 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173), b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259)
I20260619 11:09:42.667426 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cad846289304ca38e714c49f07a0125" candidate_uuid: "4093beb136354f068268a1ee70f64623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e"
I20260619 11:09:42.667529 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cad846289304ca38e714c49f07a0125" candidate_uuid: "4093beb136354f068268a1ee70f64623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a4d596f4ec4959a19b4ceba8863745"
I20260619 11:09:42.667537 8944 raft_consensus.cc:3037] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.667654 8804 raft_consensus.cc:3037] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.668799 8804 raft_consensus.cc:2454] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4093beb136354f068268a1ee70f64623 in term 1.
I20260619 11:09:42.668994 8944 raft_consensus.cc:2454] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4093beb136354f068268a1ee70f64623 in term 1.
I20260619 11:09:42.669098 9017 leader_election.cc:302] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4093beb136354f068268a1ee70f64623, b2a4d596f4ec4959a19b4ceba8863745; no voters:
I20260619 11:09:42.669324 9161 raft_consensus.cc:2781] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:42.669551 9161 raft_consensus.cc:686] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 1 LEADER]: Becoming Leader. State: Replica: 4093beb136354f068268a1ee70f64623, State: Running, Role: LEADER
I20260619 11:09:42.669863 9161 consensus_queue.cc:237] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.672932 8646 catalog_manager.cc:5495] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4093beb136354f068268a1ee70f64623 (127.7.90.67). New cstate: current_term: 1 leader_uuid: "4093beb136354f068268a1ee70f64623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } health_report { overall_health: HEALTHY } } }
I20260619 11:09:42.697748 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.693618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.7.90.67:38425)
0619 11:09:42.693739 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.693743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.693756 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:42.693958 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.693960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.694015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.694099 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:42.694347 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.694950 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.696246 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.696484 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.696595 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.696929 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.697321 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.697466 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.697558 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.697610 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:09:42.703112 8804 raft_consensus.cc:1261] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Refusing update from remote peer 4093beb136354f068268a1ee70f64623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.703112 8944 raft_consensus.cc:1261] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Refusing update from remote peer 4093beb136354f068268a1ee70f64623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.703743 9161 consensus_queue.cc:1035] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.703961 9163 consensus_queue.cc:1035] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.710531 9166 mvcc.cc:205] Tried to move back new op lower bound from 7298528799545729024 to 7298528799418384384. Current Snapshot: MvccSnapshot[applied={T|T < 7298528799545729024}]
I20260619 11:09:42.711282 9168 mvcc.cc:205] Tried to move back new op lower bound from 7298528799545729024 to 7298528799418384384. Current Snapshot: MvccSnapshot[applied={T|T < 7298528799545729024}]
I20260619 11:09:42.712334 9167 mvcc.cc:205] Tried to move back new op lower bound from 7298528799545729024 to 7298528799418384384. Current Snapshot: MvccSnapshot[applied={T|T < 7298528799545729024}]
I20260619 11:09:42.721727 8646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:42.721894 8646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:42.724383 8646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Sending DeleteTablet for 3 replicas of tablet 4cad846289304ca38e714c49f07a0125
I20260619 11:09:42.725101 8924 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cad846289304ca38e714c49f07a0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:55992
I20260619 11:09:42.725158 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cad846289304ca38e714c49f07a0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:43010
I20260619 11:09:42.725539 9064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cad846289304ca38e714c49f07a0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47660
I20260619 11:09:42.725973 9178 tablet_replica.cc:330] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: stopping tablet replica
I20260619 11:09:42.725999 9177 tablet_replica.cc:330] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: stopping tablet replica
I20260619 11:09:42.726272 9177 raft_consensus.cc:2229] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:42.726284 9178 raft_consensus.cc:2229] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:42.726231 8646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:09:42.726570 9177 raft_consensus.cc:2258] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:09:42.726619 8646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:42.726862 9178 raft_consensus.cc:2258] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:42.726895 9176 tablet_replica.cc:330] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: stopping tablet replica
I20260619 11:09:42.727125 9176 raft_consensus.cc:2229] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:42.727366 9176 raft_consensus.cc:2258] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:42.728139 9177 ts_tablet_manager.cc:1905] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.728145 9178 ts_tablet_manager.cc:1905] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.730962 9178 ts_tablet_manager.cc:1918] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:42.731119 9178 log.cc:1196] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/4cad846289304ca38e714c49f07a0125
I20260619 11:09:42.731292 9177 ts_tablet_manager.cc:1918] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:42.731419 9177 log.cc:1196] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/4cad846289304ca38e714c49f07a0125
I20260619 11:09:42.731580 9178 ts_tablet_manager.cc:1939] T 4cad846289304ca38e714c49f07a0125 P 4093beb136354f068268a1ee70f64623: Deleting consensus metadata
I20260619 11:09:42.731781 9177 ts_tablet_manager.cc:1939] T 4cad846289304ca38e714c49f07a0125 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting consensus metadata
I20260619 11:09:42.732352 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 2582af4047814f9f893cd3cb48872c13 (DEFAULT_TABLE table=test-table [id=9156c37adc6c4f6aaecb9cca34aec118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.732640 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2582af4047814f9f893cd3cb48872c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.732721 8633 catalog_manager.cc:4841] TS 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425): tablet 4cad846289304ca38e714c49f07a0125 (table test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]) successfully deleted
I20260619 11:09:42.733309 9064 tablet_service.cc:1467] Processing CreateTablet for tablet 2582af4047814f9f893cd3cb48872c13 (DEFAULT_TABLE table=test-table [id=9156c37adc6c4f6aaecb9cca34aec118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.733721 8634 catalog_manager.cc:4841] TS 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173): tablet 4cad846289304ca38e714c49f07a0125 (table test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]) successfully deleted
I20260619 11:09:42.733915 9064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2582af4047814f9f893cd3cb48872c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.734192 9176 ts_tablet_manager.cc:1905] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.735076 9154 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Bootstrap starting.
I20260619 11:09:42.735987 9155 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Bootstrap starting.
I20260619 11:09:42.736033 9154 tablet_bootstrap.cc:654] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.736474 9176 ts_tablet_manager.cc:1918] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:42.736553 9176 log.cc:1196] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/4cad846289304ca38e714c49f07a0125
I20260619 11:09:42.736833 9155 tablet_bootstrap.cc:654] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.737279 9176 ts_tablet_manager.cc:1939] T 4cad846289304ca38e714c49f07a0125 P b2a4d596f4ec4959a19b4ceba8863745: Deleting consensus metadata
I20260619 11:09:42.737457 9154 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: No bootstrap required, opened a new log
I20260619 11:09:42.737537 9154 ts_tablet_manager.cc:1397] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:42.737860 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 2582af4047814f9f893cd3cb48872c13 (DEFAULT_TABLE table=test-table [id=9156c37adc6c4f6aaecb9cca34aec118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:42.738032 9154 raft_consensus.cc:348] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.738112 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2582af4047814f9f893cd3cb48872c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.738158 9154 raft_consensus.cc:374] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.738200 9154 raft_consensus.cc:729] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a4d596f4ec4959a19b4ceba8863745, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.738327 9154 consensus_queue.cc:260] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.738582 9154 ts_tablet_manager.cc:1428] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:42.739739 8633 catalog_manager.cc:4841] TS b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259): tablet 4cad846289304ca38e714c49f07a0125 (table test-table [id=bf9382c0ab2c40119e635ac73e6c52f2]) successfully deleted
I20260619 11:09:42.740126 9153 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Bootstrap starting.
I20260619 11:09:42.741051 9153 tablet_bootstrap.cc:654] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.741822 9155 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: No bootstrap required, opened a new log
I20260619 11:09:42.741899 9155 ts_tablet_manager.cc:1397] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 11:09:42.742183 9153 tablet_bootstrap.cc:492] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: No bootstrap required, opened a new log
I20260619 11:09:42.742237 9153 ts_tablet_manager.cc:1397] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:09:42.742312 9155 raft_consensus.cc:348] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.742406 9155 raft_consensus.cc:374] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.742435 9155 raft_consensus.cc:729] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4093beb136354f068268a1ee70f64623, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.742544 9155 consensus_queue.cc:260] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.742587 9153 raft_consensus.cc:348] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.742723 9153 raft_consensus.cc:374] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.742758 9153 raft_consensus.cc:729] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa7d6a494534780b6b7dbf3c33a3e9e, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.742770 9155 ts_tablet_manager.cc:1428] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:09:42.742872 9153 consensus_queue.cc:260] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.743115 9153 ts_tablet_manager.cc:1428] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:09:42.744207 9160 raft_consensus.cc:482] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:42.744292 9160 raft_consensus.cc:504] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.745340 9160 leader_election.cc:288] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173), 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425)
I20260619 11:09:42.749698 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.745845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:57159 (local address 127.7.90.66:40173)
0619 11:09:42.745955 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.745958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.746127 (+ 169us) server_negotiation.cc:407] Connection header received
0619 11:09:42.746228 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.746231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.746308 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.746403 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.746699 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.747408 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.748530 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.749079 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.749178 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.749245 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.749468 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.749580 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:09:42.750082 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.745548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:40173 (local address 127.7.90.65:57159)
0619 11:09:42.745865 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.746040 (+ 175us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.746143 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.746415 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.746419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.746435 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.746573 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.746582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.747532 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.747535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.748412 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.748421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.749224 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.749229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.749324 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.749357 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.749815 (+ 458us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.749891 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":1280,"thread_start_us":80,"threads_started":1}
I20260619 11:09:42.750535 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.745548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:38425 (local address 127.7.90.65:34547)
0619 11:09:42.746333 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.746345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.746460 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.746772 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.746774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.746788 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.746901 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.746906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.747812 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.747816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.749035 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.749045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.750136 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.750140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.750240 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.750261 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.750329 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.750388 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":19,"thread_start_us":66,"threads_started":1}
I20260619 11:09:42.750519 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2582af4047814f9f893cd3cb48872c13" candidate_uuid: "b2a4d596f4ec4959a19b4ceba8863745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" is_pre_election: true
I20260619 11:09:42.750631 8944 raft_consensus.cc:2454] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a4d596f4ec4959a19b4ceba8863745 in term 0.
I20260619 11:09:42.750664 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.745629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:34547 (local address 127.7.90.67:38425)
0619 11:09:42.745730 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.745734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.746401 (+ 667us) server_negotiation.cc:407] Connection header received
0619 11:09:42.746566 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.746569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.746616 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.746659 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.747025 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.747701 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.749171 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.749834 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.749909 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.749972 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.750490 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.750564 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:09:42.751032 8738 leader_election.cc:302] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7d6a494534780b6b7dbf3c33a3e9e, b2a4d596f4ec4959a19b4ceba8863745; no voters:
I20260619 11:09:42.751310 9160 raft_consensus.cc:2781] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:42.751261 9084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2582af4047814f9f893cd3cb48872c13" candidate_uuid: "b2a4d596f4ec4959a19b4ceba8863745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4093beb136354f068268a1ee70f64623" is_pre_election: true
I20260619 11:09:42.751386 9160 raft_consensus.cc:482] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:42.751423 9160 raft_consensus.cc:3037] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.751511 9084 raft_consensus.cc:2454] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a4d596f4ec4959a19b4ceba8863745 in term 0.
I20260619 11:09:42.753011 9160 raft_consensus.cc:504] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.753330 9160 leader_election.cc:288] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [CANDIDATE]: Term 1 election: Requested vote from peers 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173), 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425)
I20260619 11:09:42.753671 8944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2582af4047814f9f893cd3cb48872c13" candidate_uuid: "b2a4d596f4ec4959a19b4ceba8863745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e"
I20260619 11:09:42.753758 8944 raft_consensus.cc:3037] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.753875 9084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2582af4047814f9f893cd3cb48872c13" candidate_uuid: "b2a4d596f4ec4959a19b4ceba8863745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4093beb136354f068268a1ee70f64623"
I20260619 11:09:42.753981 9084 raft_consensus.cc:3037] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.754563 8944 raft_consensus.cc:2454] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a4d596f4ec4959a19b4ceba8863745 in term 1.
I20260619 11:09:42.754846 8738 leader_election.cc:302] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7d6a494534780b6b7dbf3c33a3e9e, b2a4d596f4ec4959a19b4ceba8863745; no voters:
I20260619 11:09:42.755020 9160 raft_consensus.cc:2781] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:42.755028 9084 raft_consensus.cc:2454] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a4d596f4ec4959a19b4ceba8863745 in term 1.
I20260619 11:09:42.755239 9160 raft_consensus.cc:686] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 LEADER]: Becoming Leader. State: Replica: b2a4d596f4ec4959a19b4ceba8863745, State: Running, Role: LEADER
I20260619 11:09:42.755504 9160 consensus_queue.cc:237] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.757812 8645 catalog_manager.cc:5495] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65). New cstate: current_term: 1 leader_uuid: "b2a4d596f4ec4959a19b4ceba8863745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:42.792472 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.783561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.7.90.126:36863)
0619 11:09:42.783693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.783698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.783984 (+ 286us) server_negotiation.cc:407] Connection header received
0619 11:09:42.784099 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.784102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.784149 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.784226 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.784859 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.785736 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.786588 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.786926 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.787033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.789338 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.789340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.789343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.789363 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.790713 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.791290 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.791292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.791293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.791361 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.791625 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.791627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.791628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.791782 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.791856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.792180 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.792346 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":49}
I20260619 11:09:42.792481 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.783522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36863 (local address 127.0.0.1:40200)
0619 11:09:42.783877 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.783895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.784008 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.784264 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.784266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.784656 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:42.784747 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.784752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.785832 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.785835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.786446 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.786454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.787040 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.787043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.787154 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.787734 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:42.787745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:42.789209 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:42.790819 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.790829 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.790836 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.791191 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.791443 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:42.791446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:42.791448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:42.791542 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:42.791878 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:42.791885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:42.792017 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.792133 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.792283 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":174,"threads_started":1}
I20260619 11:09:42.796444 8645 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:09:42.797235 8645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:42.801980 8784 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd1466434f94ccf837107dd0797f794 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=890516fe5175462486bc0900cbc88121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:42.802026 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd1466434f94ccf837107dd0797f794 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=890516fe5175462486bc0900cbc88121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:42.802090 9064 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd1466434f94ccf837107dd0797f794 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=890516fe5175462486bc0900cbc88121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:42.802350 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd1466434f94ccf837107dd0797f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.802392 9064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd1466434f94ccf837107dd0797f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.802376 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd1466434f94ccf837107dd0797f794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:42.805114 9153 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Bootstrap starting.
I20260619 11:09:42.805132 9155 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: Bootstrap starting.
I20260619 11:09:42.805315 9154 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: Bootstrap starting.
I20260619 11:09:42.806239 9154 tablet_bootstrap.cc:654] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.806519 9155 tablet_bootstrap.cc:654] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.806519 9153 tablet_bootstrap.cc:654] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:42.807885 9155 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: No bootstrap required, opened a new log
I20260619 11:09:42.807997 9155 ts_tablet_manager.cc:1397] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:09:42.808157 9153 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: No bootstrap required, opened a new log
I20260619 11:09:42.808238 9153 ts_tablet_manager.cc:1397] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
W20260619 11:09:42.808422 9131 tablet.cc:2369] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:42.808449 9155 raft_consensus.cc:348] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.808544 9155 raft_consensus.cc:374] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.808573 9155 raft_consensus.cc:729] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4093beb136354f068268a1ee70f64623, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.808632 9153 raft_consensus.cc:348] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.808679 9155 consensus_queue.cc:260] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.808728 9153 raft_consensus.cc:374] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.808758 9153 raft_consensus.cc:729] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa7d6a494534780b6b7dbf3c33a3e9e, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.808862 9153 consensus_queue.cc:260] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.808899 9155 ts_tablet_manager.cc:1428] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:09:42.808964 9173 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.809083 9173 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.809100 9153 ts_tablet_manager.cc:1428] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:42.809268 9159 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.809384 9159 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.810655 9154 tablet_bootstrap.cc:492] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: No bootstrap required, opened a new log
I20260619 11:09:42.810735 9154 ts_tablet_manager.cc:1397] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 11:09:42.811137 9154 raft_consensus.cc:348] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.811232 9154 raft_consensus.cc:374] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:42.811266 9154 raft_consensus.cc:729] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a4d596f4ec4959a19b4ceba8863745, State: Initialized, Role: FOLLOWER
I20260619 11:09:42.811379 9154 consensus_queue.cc:260] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.811601 9160 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.811738 9160 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.811619 9154 ts_tablet_manager.cc:1428] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:09:42.816378 9159 raft_consensus.cc:482] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:42.816490 9159 raft_consensus.cc:504] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.817473 9159 leader_election.cc:288] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259), 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425)
I20260619 11:09:42.821803 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.817713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:59519 (local address 127.7.90.65:35259)
0619 11:09:42.817821 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.817825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.817855 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:09:42.818032 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.818036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.818080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.818148 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.818499 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.819280 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.820766 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.821262 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.821377 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.821435 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.821625 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.821692 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:09:42.821805 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.817657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35259 (local address 127.7.90.66:59519)
0619 11:09:42.817783 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.817801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.817877 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.818259 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.818264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.818283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.818398 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.818404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.819415 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.819419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.820633 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.820643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.821374 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.821377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.821481 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.821502 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.821610 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.821685 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":10}
I20260619 11:09:42.822220 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd1466434f94ccf837107dd0797f794" candidate_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a4d596f4ec4959a19b4ceba8863745" is_pre_election: true
I20260619 11:09:42.822361 8804 raft_consensus.cc:2454] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa7d6a494534780b6b7dbf3c33a3e9e in term 0.
I20260619 11:09:42.822695 8877 leader_election.cc:302] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7d6a494534780b6b7dbf3c33a3e9e, b2a4d596f4ec4959a19b4ceba8863745; no voters:
I20260619 11:09:42.822930 9159 raft_consensus.cc:2781] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:42.823001 9159 raft_consensus.cc:482] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:42.823069 9159 raft_consensus.cc:3037] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.823194 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.817892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:38425 (local address 127.7.90.66:50731)
0619 11:09:42.818508 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.818520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.818632 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.818856 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.818859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.818873 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:42.818986 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.818991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.819680 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.819685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.820954 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.820964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.822663 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.822669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.822765 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.822826 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.822897 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.822990 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":90,"threads_started":1}
I20260619 11:09:42.824043 9159 raft_consensus.cc:504] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.824294 9159 leader_election.cc:288] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [CANDIDATE]: Term 1 election: Requested vote from peers b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259), 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425)
I20260619 11:09:42.824652 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd1466434f94ccf837107dd0797f794" candidate_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a4d596f4ec4959a19b4ceba8863745"
I20260619 11:09:42.824728 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.818218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:50731 (local address 127.7.90.67:38425)
0619 11:09:42.818322 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.818326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.818601 (+ 275us) server_negotiation.cc:407] Connection header received
0619 11:09:42.818717 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.818720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.818762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.818836 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:42.819087 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.819571 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.821077 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.821698 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.821792 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.821843 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.824530 (+ 2687us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.824610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:09:42.824761 8804 raft_consensus.cc:3037] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.825608 8804 raft_consensus.cc:2454] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa7d6a494534780b6b7dbf3c33a3e9e in term 1.
I20260619 11:09:42.825868 8877 leader_election.cc:302] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7d6a494534780b6b7dbf3c33a3e9e, b2a4d596f4ec4959a19b4ceba8863745; no voters:
I20260619 11:09:42.826048 9084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd1466434f94ccf837107dd0797f794" candidate_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4093beb136354f068268a1ee70f64623" is_pre_election: true
I20260619 11:09:42.826090 9159 raft_consensus.cc:2781] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:42.826136 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd1466434f94ccf837107dd0797f794" candidate_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4093beb136354f068268a1ee70f64623"
I20260619 11:09:42.826170 9084 raft_consensus.cc:2454] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa7d6a494534780b6b7dbf3c33a3e9e in term 0.
I20260619 11:09:42.826284 9159 raft_consensus.cc:686] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 LEADER]: Becoming Leader. State: Replica: 0fa7d6a494534780b6b7dbf3c33a3e9e, State: Running, Role: LEADER
I20260619 11:09:42.826531 9159 consensus_queue.cc:237] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } }
I20260619 11:09:42.828276 9196 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: TxnStatusTablet state changed. Reason: New leader 0fa7d6a494534780b6b7dbf3c33a3e9e. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.828433 9196 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:42.828734 9198 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:09:42.828718 8646 catalog_manager.cc:5495] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66). New cstate: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } health_report { overall_health: UNKNOWN } } }
I20260619 11:09:42.828920 9198 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260619 11:09:42.834352 8991 tablet.cc:2369] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:42.844810 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.839947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.7.90.66:40173)
0619 11:09:42.840057 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.840062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.840075 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:42.840157 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.840160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.840219 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.840322 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:42.840649 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.841456 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.842752 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.843091 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.843192 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.843520 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.844367 (+ 847us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.844476 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.844595 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.844646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:42.844810 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.839663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:40173 (local address 127.0.0.1:56004)
0619 11:09:42.839817 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.839833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.839934 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.840347 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.840351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.840362 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:42.840488 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.840497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.841661 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.841665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.842612 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.842623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.843209 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.843213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.843315 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.843364 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.844473 (+ 1109us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.844488 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.844564 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.844649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 11:09:42.847853 9083 raft_consensus.cc:3037] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:42.847878 8804 raft_consensus.cc:1261] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Refusing update from remote peer 0fa7d6a494534780b6b7dbf3c33a3e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.849252 9083 raft_consensus.cc:1261] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Refusing update from remote peer 0fa7d6a494534780b6b7dbf3c33a3e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.849550 9159 consensus_queue.cc:1035] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.850155 9159 consensus_queue.cc:1035] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.851183 9160 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: TxnStatusTablet state changed. Reason: New leader 0fa7d6a494534780b6b7dbf3c33a3e9e. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.851296 9160 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.851999 9173 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: TxnStatusTablet state changed. Reason: New leader 0fa7d6a494534780b6b7dbf3c33a3e9e. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.852123 9173 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.854806 9159 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.854908 9159 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:42.855654 9196 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.855747 9196 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 0fa7d6a494534780b6b7dbf3c33a3e9e: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:42.856477 9160 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.856578 9160 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P b2a4d596f4ec4959a19b4ceba8863745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:42.857245 9173 tablet_replica.cc:439] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a4d596f4ec4959a19b4ceba8863745" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35259 } } peers { permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 } } peers { permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 } } }
I20260619 11:09:42.857334 9173 tablet_replica.cc:442] T 2bd1466434f94ccf837107dd0797f794 P 4093beb136354f068268a1ee70f64623: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 11:09:42.870187 8851 tablet.cc:2369] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:42.871102 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.866599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.7.90.65:35259)
0619 11:09:42.866706 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.866710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.866722 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:42.866792 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.866795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.866850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.866945 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:42.867286 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.868047 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.869191 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.869569 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.869659 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.869990 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.870468 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.870568 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.870913 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.870967 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:42.894163 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.881602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:38729 (local address 127.7.90.66:40173)
0619 11:09:42.881708 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.881713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.889838 (+ 8125us) server_negotiation.cc:407] Connection header received
0619 11:09:42.889962 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.889966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.890016 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.890095 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:42.890354 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.891103 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.892905 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.893239 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.893345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.893693 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.893778 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.893852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.893972 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.894025 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:09:42.894189 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.881509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:40173 (local address 127.7.90.65:38729)
0619 11:09:42.889733 (+ 8224us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.889748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.889857 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.890117 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.890120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.890127 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:42.890246 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.890252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.891213 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.891216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.892765 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.892776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.893358 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.893362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.893464 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.893537 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.893872 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.893878 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.893948 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.894026 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8102,"thread_start_us":92,"threads_started":1}
I20260619 11:09:42.901712 9083 raft_consensus.cc:1261] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Refusing update from remote peer b2a4d596f4ec4959a19b4ceba8863745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.901785 8944 raft_consensus.cc:1261] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Refusing update from remote peer b2a4d596f4ec4959a19b4ceba8863745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:42.902144 9184 consensus_queue.cc:1035] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4093beb136354f068268a1ee70f64623" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.902402 9160 consensus_queue.cc:1035] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa7d6a494534780b6b7dbf3c33a3e9e" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:42.929339 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.925565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:51153 (local address 127.7.90.65:35259)
0619 11:09:42.925667 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.925672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.925922 (+ 250us) server_negotiation.cc:407] Connection header received
0619 11:09:42.926108 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.926111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.926157 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.926200 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:42.926591 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.927104 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.928353 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.928589 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.928674 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.928977 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.929039 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.929087 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.929194 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.929239 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:09:42.929440 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.925419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35259 (local address 127.7.90.66:51153)
0619 11:09:42.925846 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:42.925860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:42.925997 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:42.926327 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:42.926329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:42.926337 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:42.926463 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.926469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.927205 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.927209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.928225 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:42.928235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.928696 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.928699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:42.928797 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.928850 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:42.929113 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:42.929119 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:09:42.929185 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:09:42.929261 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260619 11:09:42.952054 8645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:42.952221 8645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:42.954891 8645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40f55735d960431080b337f526598ccd: Sending DeleteTablet for 3 replicas of tablet 2582af4047814f9f893cd3cb48872c13
I20260619 11:09:42.955507 9064 tablet_service.cc:1514] Processing DeleteTablet for tablet 2582af4047814f9f893cd3cb48872c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47660
I20260619 11:09:42.955696 9178 tablet_replica.cc:330] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: stopping tablet replica
I20260619 11:09:42.955717 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2582af4047814f9f893cd3cb48872c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:43010
I20260619 11:09:42.955807 9178 raft_consensus.cc:2229] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:42.955935 9176 tablet_replica.cc:330] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: stopping tablet replica
I20260619 11:09:42.955973 9178 raft_consensus.cc:2258] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:42.956041 9176 raft_consensus.cc:2229] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:42.956295 9176 raft_consensus.cc:2258] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:42.956832 9178 ts_tablet_manager.cc:1905] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.957067 9176 ts_tablet_manager.cc:1905] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.957053 8924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2582af4047814f9f893cd3cb48872c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:42 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:55992
I20260619 11:09:42.957219 9177 tablet_replica.cc:330] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: stopping tablet replica
I20260619 11:09:42.957322 9177 raft_consensus.cc:2229] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:42.957417 9177 raft_consensus.cc:2258] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:42.958844 9177 ts_tablet_manager.cc:1905] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:42.959652 9176 ts_tablet_manager.cc:1918] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:42.959652 9178 ts_tablet_manager.cc:1918] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:42.959720 9178 log.cc:1196] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/2582af4047814f9f893cd3cb48872c13
I20260619 11:09:42.959719 9176 log.cc:1196] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/2582af4047814f9f893cd3cb48872c13
I20260619 11:09:42.959995 9176 ts_tablet_manager.cc:1939] T 2582af4047814f9f893cd3cb48872c13 P b2a4d596f4ec4959a19b4ceba8863745: Deleting consensus metadata
I20260619 11:09:42.960012 9178 ts_tablet_manager.cc:1939] T 2582af4047814f9f893cd3cb48872c13 P 4093beb136354f068268a1ee70f64623: Deleting consensus metadata
I20260619 11:09:42.960608 8633 catalog_manager.cc:4841] TS b2a4d596f4ec4959a19b4ceba8863745 (127.7.90.65:35259): tablet 2582af4047814f9f893cd3cb48872c13 (table test-table [id=9156c37adc6c4f6aaecb9cca34aec118]) successfully deleted
I20260619 11:09:42.960733 8633 catalog_manager.cc:4841] TS 4093beb136354f068268a1ee70f64623 (127.7.90.67:38425): tablet 2582af4047814f9f893cd3cb48872c13 (table test-table [id=9156c37adc6c4f6aaecb9cca34aec118]) successfully deleted
I20260619 11:09:42.961392 9177 ts_tablet_manager.cc:1918] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:42.961472 9177 log.cc:1196] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/2582af4047814f9f893cd3cb48872c13
I20260619 11:09:42.961742 9177 ts_tablet_manager.cc:1939] T 2582af4047814f9f893cd3cb48872c13 P 0fa7d6a494534780b6b7dbf3c33a3e9e: Deleting consensus metadata
I20260619 11:09:42.962307 8634 catalog_manager.cc:4841] TS 0fa7d6a494534780b6b7dbf3c33a3e9e (127.7.90.66:40173): tablet 2582af4047814f9f893cd3cb48872c13 (table test-table [id=9156c37adc6c4f6aaecb9cca34aec118]) successfully deleted
Jun 19 11:09:42 dist-test-slave-f1t5 krb5kdc[8602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:42.973855 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.957843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.7.90.65:35259)
0619 11:09:42.957939 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.957944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.961822 (+ 3878us) server_negotiation.cc:407] Connection header received
0619 11:09:42.962012 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.962015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.962060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.962109 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.963075 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.963890 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.965153 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.965508 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.965646 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.969307 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.969316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.969330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.969354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.971765 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.972253 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.972259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.972261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.972344 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.972658 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.972661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.972663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.972934 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.973108 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.973551 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.973689 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 11:09:42.974510 8844 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43034
I20260619 11:09:42.985394 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:42.976278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.7.90.126:36863)
0619 11:09:42.976409 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:42.976413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:42.976490 (+ 77us) server_negotiation.cc:407] Connection header received
0619 11:09:42.976664 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:42.976667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:42.976716 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:42.976801 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:42.977574 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.978447 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.979578 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:42.979910 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:42.980027 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:42.981481 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:42.981485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:42.981488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:42.981516 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:42.983265 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.983689 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.983693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.983695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.983786 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:42.984094 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:42.984097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:42.984099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:42.984319 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:42.984472 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:42.985118 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 11:09:42.985235 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:09:42.986140 8645 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40202
I20260619 11:09:42.987771 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8719
I20260619 11:09:42.994653 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8859
I20260619 11:09:43.000367 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8999
I20260619 11:09:43.006464 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 8615
2026-06-19T11:09:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2462 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:43 dist-test-slave-f1t5 krb5kdc[9220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:43 dist-test-slave-f1t5 krb5kdc[9220](info): set up 2 sockets
Jun 19 11:09:43 dist-test-slave-f1t5 krb5kdc[9220](info): commencing operation
krb5kdc: starting...
W20260619 11:09:44.033179 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:44.146508 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33287
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:44.252815 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:44.253095 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:44.253177 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:44.257123 9233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:44.257203 9233 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:09:44.257236 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:44.257298 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:44.257352 9233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:44.257387 9233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:44.262578 9233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:33287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:44.263568 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:44.264537 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:44.272550 9239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:44.272553 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:44.272615 9238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.272801 9233 server_base.cc:1034] running on GCE node
I20260619 11:09:44.273319 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:44.274430 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:44.275594 9233 hybrid_clock.cc:648] HybridClock initialized: now 1781867384275565 us; error 39 us; skew 500 ppm
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:44.278168 9233 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:44.278919 9233 webserver.cc:460] Webserver started at http://127.7.90.126:43405/ using document root <none> and password file <none>
I20260619 11:09:44.279457 9233 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:44.279542 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:44.279754 9233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:44.281737 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "b15dff3286224f0ca8828cb255cfe6c9"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "c62d28abc1d17e5daaaafc436c8e05f0"
server_key_iv: "b12fb9e8c1767001932362c1e49d128d"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.282280 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "b15dff3286224f0ca8828cb255cfe6c9"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "c62d28abc1d17e5daaaafc436c8e05f0"
server_key_iv: "b12fb9e8c1767001932362c1e49d128d"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.285482 9233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:44.287706 9248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.288827 9233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:44.288957 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "b15dff3286224f0ca8828cb255cfe6c9"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "c62d28abc1d17e5daaaafc436c8e05f0"
server_key_iv: "b12fb9e8c1767001932362c1e49d128d"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.289065 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:44.306449 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:44.308928 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:44.309109 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:44.316444 9233 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:33287
I20260619 11:09:44.316449 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:33287 every 8 connection(s)
I20260619 11:09:44.317365 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:44.320205 9311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:44.322106 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 9233
I20260619 11:09:44.322255 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:44.322530 7529 external_mini_cluster.cc:1428] Setting key ec070281ebfb54778080d66946a42fda
I20260619 11:09:44.326175 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Bootstrap starting.
I20260619 11:09:44.328678 9311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:44.329483 9311 log.cc:824] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:44.331691 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: No bootstrap required, opened a new log
I20260619 11:09:44.335208 9311 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } }
I20260619 11:09:44.335457 9311 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:44.335518 9311 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15dff3286224f0ca8828cb255cfe6c9, State: Initialized, Role: FOLLOWER
I20260619 11:09:44.336030 9311 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } }
I20260619 11:09:44.336182 9311 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:44.336242 9311 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:44.336330 9311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:44.337019 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.323904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.7.90.126:33287)
0619 11:09:44.324314 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:44.324330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:44.324373 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:09:44.325206 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:44.325226 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:44.325627 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:44.325904 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:44.326757 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.327748 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.328937 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.329392 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.329553 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.332283 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:44.332287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:44.332301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:44.332326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:44.334032 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.334471 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.334476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.334481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.334585 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.334869 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.334873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.334874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.335242 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:44.335471 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:44.335774 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 11:09:44.336057 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260619 11:09:44.337383 9311 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } }
I20260619 11:09:44.337752 9311 leader_election.cc:302] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15dff3286224f0ca8828cb255cfe6c9; no voters:
I20260619 11:09:44.338104 9311 leader_election.cc:288] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:44.338465 9317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:44.338639 9317 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [term 1 LEADER]: Becoming Leader. State: Replica: b15dff3286224f0ca8828cb255cfe6c9, State: Running, Role: LEADER
I20260619 11:09:44.338995 9317 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } }
I20260619 11:09:44.339383 9311 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:44.340148 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15dff3286224f0ca8828cb255cfe6c9. Latest consensus state: current_term: 1 leader_uuid: "b15dff3286224f0ca8828cb255cfe6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } } }
I20260619 11:09:44.340276 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:44.340934 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15dff3286224f0ca8828cb255cfe6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15dff3286224f0ca8828cb255cfe6c9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33287 } } }
I20260619 11:09:44.342079 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:44.342163 9325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:44.345400 9325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:44.351122 9325 catalog_manager.cc:1347] Generated new cluster ID: a63c78033f3940ae99af22f3eb634b08
I20260619 11:09:44.351195 9325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:44.361373 9325 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:44.362666 9325 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:44.367453 9325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Generated new TSK 0
I20260619 11:09:44.368108 9325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:44.403811 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33287
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:44.528918 9337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:44.529201 9337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:44.529277 9337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:44.533151 9337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:44.533234 9337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:44.533306 9337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:09:44.533351 9337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:44.533419 9337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:44.538462 9337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:44.539610 9337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:44.540712 9337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:44.549880 9345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:44.549896 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.550076 9337 server_base.cc:1034] running on GCE node
W20260619 11:09:44.549930 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.550588 9337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:44.551180 9337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:44.552343 9337 hybrid_clock.cc:648] HybridClock initialized: now 1781867384552326 us; error 29 us; skew 500 ppm
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:44.555130 9337 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:44.555954 9337 webserver.cc:460] Webserver started at http://127.7.90.65:38779/ using document root <none> and password file <none>
I20260619 11:09:44.556526 9337 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:44.556624 9337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:44.556859 9337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:44.558979 9337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "174dc4d25230493391975a9db90a88d2"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "1118c15517938c843712a7c8183f2fe7"
server_key_iv: "24103c417022f90a227aba0b94dfa6a8"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.559613 9337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "174dc4d25230493391975a9db90a88d2"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "1118c15517938c843712a7c8183f2fe7"
server_key_iv: "24103c417022f90a227aba0b94dfa6a8"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.563320 9337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:44.565742 9352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.566936 9337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:44.567070 9337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "174dc4d25230493391975a9db90a88d2"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "1118c15517938c843712a7c8183f2fe7"
server_key_iv: "24103c417022f90a227aba0b94dfa6a8"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.567174 9337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:44.581907 9337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:44.584687 9337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:44.584884 9337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:44.586306 9337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:44.586377 9337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.586445 9337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:44.586494 9337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:44.597047 9337 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:38439
I20260619 11:09:44.597117 9467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:38439 every 8 connection(s)
I20260619 11:09:44.598057 9337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:44.600078 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.587148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:56861 (local address 127.7.90.126:33287)
0619 11:09:44.587268 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:44.587271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:44.588289 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 11:09:44.589392 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:44.589396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:44.589480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:44.589572 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:44.590808 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.591321 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.592663 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.592941 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.593055 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.596557 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:44.596563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:44.596570 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:44.596594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:44.598212 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.598794 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.598798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.598800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.598885 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.599200 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.599203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.599204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.599368 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:44.599449 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:44.599859 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:09:44.599964 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:44.600358 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 9337
I20260619 11:09:44.600448 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:44.600660 7529 external_mini_cluster.cc:1428] Setting key 3b32eb7f3db9a6ae1d388de2321505cd
I20260619 11:09:44.600850 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.587700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.65:56861)
0619 11:09:44.588149 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:44.588188 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:44.589112 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:44.589832 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:44.589839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:44.590219 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:44.590665 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.590677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.591454 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.591459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.592546 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.592554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.593104 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.593108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.593332 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.594004 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:44.594029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:44.596379 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:44.598329 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.598335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.598349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.598674 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.598968 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.598970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.598972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.599113 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.599453 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:44.599458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:44.599669 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 11:09:44.599817 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:09:44.600058 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260619 11:09:44.602262 9468 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33287
I20260619 11:09:44.602385 9468 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:44.602799 9468 heartbeater.cc:511] Master 127.7.90.126:33287 requested a full tablet report, sending...
I20260619 11:09:44.604666 9265 ts_manager.cc:194] Registered new tserver with Master: 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439)
I20260619 11:09:44.606009 9265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:56861
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:09:44.613091 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.605648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:35543 (local address 127.7.90.126:33287)
0619 11:09:44.605758 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:44.605762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:44.605773 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:09:44.605800 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:44.605802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:44.605835 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:44.605924 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:44.606399 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.607082 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.608242 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.608685 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.608816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.610017 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:44.610021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:44.610025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:44.610052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:44.611618 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.612035 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.612038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.612040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.612109 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.612341 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.612343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.612344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.612505 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:44.612579 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:44.612824 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 11:09:44.612932 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:44.613103 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.604543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.65:35543)
0619 11:09:44.604822 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:44.604840 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:44.604932 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:44.605939 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:44.605942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:44.606176 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:44.606293 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.606299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.607236 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.607239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.608110 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.608120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.608816 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.608820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.608904 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.609485 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:44.609498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:44.609879 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:44.611737 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.611740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.611742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.611928 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.612190 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.612192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.612193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.612265 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.612583 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:44.612585 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:44.612654 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 11:09:44.612765 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:09:44.612899 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:44.628636 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33287
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:44.735035 9477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:44.735280 9477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:44.735327 9477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:44.739166 9477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:44.739219 9477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:44.739260 9477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:09:44.739276 9477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:44.739316 9477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:44.744383 9477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:44.745247 9477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:44.746217 9477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:44.755577 9483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.755689 9477 server_base.cc:1034] running on GCE node
W20260619 11:09:44.755584 9482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:44.755636 9485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.756366 9477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:44.756950 9477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:44.758102 9477 hybrid_clock.cc:648] HybridClock initialized: now 1781867384758083 us; error 29 us; skew 500 ppm
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:44.760807 9477 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:44.762099 9477 webserver.cc:460] Webserver started at http://127.7.90.66:33367/ using document root <none> and password file <none>
I20260619 11:09:44.762715 9477 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:44.762812 9477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:44.763063 9477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:44.765214 9477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "af0803920cb44a72a2186fcf7ef67979"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "b365fbfe76186aec2c07a5556bc65c90"
server_key_iv: "34aca064c6900377a66e327b71602124"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.765794 9477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "af0803920cb44a72a2186fcf7ef67979"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "b365fbfe76186aec2c07a5556bc65c90"
server_key_iv: "34aca064c6900377a66e327b71602124"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.769670 9477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 11:09:44.772403 9492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.773731 9477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:44.773854 9477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "af0803920cb44a72a2186fcf7ef67979"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "b365fbfe76186aec2c07a5556bc65c90"
server_key_iv: "34aca064c6900377a66e327b71602124"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.773973 9477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:44.796741 9477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:44.799568 9477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:44.799788 9477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:44.801216 9477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:44.801287 9477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.801358 9477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:44.801394 9477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:44.812110 9477 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:43409
I20260619 11:09:44.812191 9607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:43409 every 8 connection(s)
I20260619 11:09:44.813216 9477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:44.814471 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 9477
I20260619 11:09:44.814605 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:44.814869 7529 external_mini_cluster.cc:1428] Setting key 994fd1d45c3240c6062d8f7f41ec76ba
I20260619 11:09:44.815780 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.801986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:38747 (local address 127.7.90.126:33287)
0619 11:09:44.802121 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:44.802125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:44.803716 (+ 1591us) server_negotiation.cc:407] Connection header received
0619 11:09:44.804871 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:44.804875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:44.804936 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:44.805052 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:44.806738 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.807387 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.809139 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.809342 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.809416 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.812104 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:44.812108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:44.812111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:44.812138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:44.813661 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.814255 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.814258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.814260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.814356 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.814716 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.814719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.814721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.814899 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:44.815018 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:44.815506 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 11:09:44.815616 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:09:44.816545 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.803050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.66:38747)
0619 11:09:44.803562 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:44.803604 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:44.804653 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:44.805240 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:44.805252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:44.805818 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:44.806561 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.806584 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.807490 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.807495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.809005 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.809017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.809509 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.809540 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.809748 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.810347 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:44.810361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:44.811929 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:44.813789 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.813794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.813808 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.814140 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.814460 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.814462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.814463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.814615 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.815072 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:44.815076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:44.815274 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 11:09:44.815454 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 11:09:44.815736 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260619 11:09:44.817842 9608 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33287
I20260619 11:09:44.818001 9608 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:44.818456 9608 heartbeater.cc:511] Master 127.7.90.126:33287 requested a full tablet report, sending...
I20260619 11:09:44.819480 9263 ts_manager.cc:194] Registered new tserver with Master: af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409)
I20260619 11:09:44.820231 9263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:38747
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:09:44.827787 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.819976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.66:44697)
0619 11:09:44.820369 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:44.820384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:44.820579 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:44.820819 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:44.820822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:44.821039 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:44.821187 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.821198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.822189 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.822193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.823153 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:44.823165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.823583 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.823589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:44.823694 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.824140 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:44.824160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:44.824590 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:44.826327 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.826329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.826331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.826506 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.826759 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:44.826761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:44.826762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:44.826859 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:44.827215 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:44.827222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:44.827316 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 11:09:44.827429 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:09:44.827590 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260619 11:09:44.827797 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:44.820037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:44697 (local address 127.7.90.126:33287)
0619 11:09:44.820173 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:44.820179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:44.820445 (+ 266us) server_negotiation.cc:407] Connection header received
0619 11:09:44.820685 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:44.820688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:44.820728 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:44.820801 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:44.821322 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.822053 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.823272 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:44.823470 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:44.823544 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:44.824723 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:44.824726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:44.824728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:44.824747 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:44.826239 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.826588 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.826590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.826592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.826656 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:44.826932 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:44.826934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:44.826935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:44.827119 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:44.827190 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:44.827541 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 11:09:44.827625 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:44.842216 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33287
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:44.949877 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:44.950137 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:44.950207 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:44.954327 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:44.954412 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:44.954487 9617 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:09:44.954530 9617 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:09:44.954591 9617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:44.959794 9617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:44.960891 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:44.962009 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:44.971553 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:44.971577 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.971612 9617 server_base.cc:1034] running on GCE node
W20260619 11:09:44.971563 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:44.972141 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:44.972757 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:44.973906 9617 hybrid_clock.cc:648] HybridClock initialized: now 1781867384973893 us; error 25 us; skew 500 ppm
Jun 19 11:09:44 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:44.976635 9617 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:44.977509 9617 webserver.cc:460] Webserver started at http://127.7.90.67:43273/ using document root <none> and password file <none>
I20260619 11:09:44.978142 9617 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:44.978227 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:44.978432 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:44.980392 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "20d976bf731741c1a36632b5e16e83cb"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "5f47131929fe4af9ec517c59615fcd3d"
server_key_iv: "b3c295eb8806e20b3bfe92e401fc3fc1"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.980976 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "20d976bf731741c1a36632b5e16e83cb"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "5f47131929fe4af9ec517c59615fcd3d"
server_key_iv: "b3c295eb8806e20b3bfe92e401fc3fc1"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.984498 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:44.986881 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:44.988041 9617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:44.988173 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "20d976bf731741c1a36632b5e16e83cb"
format_stamp: "Formatted at 2026-06-19 11:09:44 on dist-test-slave-f1t5"
server_key: "5f47131929fe4af9ec517c59615fcd3d"
server_key_iv: "b3c295eb8806e20b3bfe92e401fc3fc1"
server_key_version: "encryptionkey@0"
I20260619 11:09:44.988279 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:45.000196 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:45.003525 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:45.003720 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:45.005064 9617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:45.005138 9617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:45.005209 9617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:45.005252 9617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:09:45 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:45.016026 9617 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:46117
I20260619 11:09:45.016187 9747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:46117 every 8 connection(s)
I20260619 11:09:45.017047 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:45.017935 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 9617
I20260619 11:09:45.018115 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:45.018399 7529 external_mini_cluster.cc:1428] Setting key 756d393303d460d3c67b56734b75e717
I20260619 11:09:45.018939 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.005908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:39815 (local address 127.7.90.126:33287)
0619 11:09:45.006067 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.006072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.007304 (+ 1232us) server_negotiation.cc:407] Connection header received
0619 11:09:45.008339 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.008342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.008400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.008506 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:45.009998 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.010510 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.011637 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.011899 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.012024 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.014937 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:45.014944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:45.014947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:45.014978 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:45.016588 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.017182 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.017186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.017188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.017277 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.017707 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.017713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.017714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.017975 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:45.018082 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.018644 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.018769 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:09:45.019791 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.006699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.67:39815)
0619 11:09:45.007136 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.007187 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.008125 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.008681 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.008691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.009140 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:45.009828 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.009843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.010630 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.010635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.011509 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.011518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.012068 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.012072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.012243 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.012809 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:45.012831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:45.014741 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:45.016695 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.016703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.016719 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.017049 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.017380 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.017385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.017387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.017591 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.018130 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:45.018136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:45.018387 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.018572 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.018898 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260619 11:09:45.020905 9748 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33287
I20260619 11:09:45.021034 9748 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:45.021574 9748 heartbeater.cc:511] Master 127.7.90.126:33287 requested a full tablet report, sending...
I20260619 11:09:45.022590 9263 ts_manager.cc:194] Registered new tserver with Master: 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.023345 9263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:39815
I20260619 11:09:45.029688 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.023117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.7.90.67:42241)
0619 11:09:45.023441 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.023475 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.023589 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.023785 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.023787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.023947 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:45.024034 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.024039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.024971 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.024973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.025613 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.025628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.026015 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.026018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.026082 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.026480 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:45.026490 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:45.026762 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:45.028441 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.028443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.028444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.028620 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.028875 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.028877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.028878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.028948 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.029251 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:45.029253 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:45.029319 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.029393 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.029487 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":85,"threads_started":1}
I20260619 11:09:45.029695 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.023192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:42241 (local address 127.7.90.126:33287)
0619 11:09:45.023316 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.023319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.023533 (+ 214us) server_negotiation.cc:407] Connection header received
0619 11:09:45.023673 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.023675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.023707 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.023768 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:45.024117 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.024847 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.025716 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.025918 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.026000 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.026875 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:45.026878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:45.026880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:45.026900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:45.028342 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.028714 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.028717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.028718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.028787 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.029031 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.029033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.029034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.029175 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:45.029246 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.029434 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.029584 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:09:45.032146 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:09:45 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:09:45 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:45.056255 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.048639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.7.90.126:33287)
0619 11:09:45.048764 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.048767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.048779 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:45.048813 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.048815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.048850 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.048939 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:45.049651 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.050178 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.051038 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.051282 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.051390 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.053787 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:45.053790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:45.053792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:45.053813 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:45.054949 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.055302 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.055304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.055305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.055374 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.055596 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.055598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.055599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.055750 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:45.055848 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.056031 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.056119 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:09:45.059096 9263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:45.062156 9263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:45.076045 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.070278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.7.90.66:43409)
0619 11:09:45.070610 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.070676 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.070702 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:09:45.071399 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.071405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.071691 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.071869 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.072123 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.073070 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.074757 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.075392 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.075514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.075677 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.075789 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.075875 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260619 11:09:45.076035 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.070176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.0.0.1:40830)
0619 11:09:45.070514 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.070562 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.070680 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.071859 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.071866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.071891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.072016 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.072023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.073203 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.073206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.074628 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.074638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.075514 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.075517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.075613 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.075646 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.075761 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.075871 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":20,"thread_start_us":79,"threads_started":1}
I20260619 11:09:45.076414 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.070933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:38439 (local address 127.0.0.1:47926)
0619 11:09:45.071216 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.071229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.071314 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.072292 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.072295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.072323 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.072423 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.072443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.073709 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.073713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.074834 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.074842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.075838 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.075842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.075939 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.075962 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.076025 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.076087 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260619 11:09:45.077132 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.071323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.7.90.65:38439)
0619 11:09:45.071837 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.071851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.071872 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:09:45.071948 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.071952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.072146 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.072309 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.072562 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.073580 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.074969 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.075716 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.076678 (+ 962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.076837 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.076885 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.076989 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":115,"threads_started":1}
I20260619 11:09:45.077939 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.070936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:46117 (local address 127.0.0.1:46176)
0619 11:09:45.072548 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.072560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.072650 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.073046 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.073049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.073066 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.073153 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.073159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.074487 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.074490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.075438 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.075448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.077556 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.077559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.077653 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.077674 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.077743 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.077807 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":42,"thread_start_us":64,"threads_started":1}
I20260619 11:09:45.078184 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.071084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.7.90.67:46117)
0619 11:09:45.071609 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.071620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.072655 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 11:09:45.072736 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.072741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.072899 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.073046 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.073280 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.074359 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.075570 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.076512 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.076612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.076770 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.077980 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.078074 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":108,"threads_started":1}
I20260619 11:09:45.079077 9542 tablet_service.cc:1467] Processing CreateTablet for tablet dd9799c0b533406687b9d064469d6561 (DEFAULT_TABLE table=test-table [id=3910dbe438784967bd5c10e435345c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.079677 9402 tablet_service.cc:1467] Processing CreateTablet for tablet dd9799c0b533406687b9d064469d6561 (DEFAULT_TABLE table=test-table [id=3910dbe438784967bd5c10e435345c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.080173 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9799c0b533406687b9d064469d6561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.080674 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9799c0b533406687b9d064469d6561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.081241 9682 tablet_service.cc:1467] Processing CreateTablet for tablet dd9799c0b533406687b9d064469d6561 (DEFAULT_TABLE table=test-table [id=3910dbe438784967bd5c10e435345c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.082141 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9799c0b533406687b9d064469d6561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.087687 9771 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Bootstrap starting.
I20260619 11:09:45.088533 9772 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Bootstrap starting.
I20260619 11:09:45.089835 9771 tablet_bootstrap.cc:654] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.089846 9773 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Bootstrap starting.
I20260619 11:09:45.090404 9772 tablet_bootstrap.cc:654] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.090608 9771 log.cc:824] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:45.091223 9772 log.cc:824] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:45.091835 9773 tablet_bootstrap.cc:654] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.092571 9773 log.cc:824] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:45.092928 9771 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: No bootstrap required, opened a new log
I20260619 11:09:45.093118 9771 ts_tablet_manager.cc:1397] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:09:45.093940 9772 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: No bootstrap required, opened a new log
I20260619 11:09:45.094139 9772 ts_tablet_manager.cc:1397] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:09:45.094595 9773 tablet_bootstrap.cc:492] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: No bootstrap required, opened a new log
I20260619 11:09:45.094759 9773 ts_tablet_manager.cc:1397] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:09:45.096848 9772 raft_consensus.cc:348] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.097079 9772 raft_consensus.cc:374] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.096904 9771 raft_consensus.cc:348] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.097132 9772 raft_consensus.cc:729] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.097157 9771 raft_consensus.cc:374] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.097203 9771 raft_consensus.cc:729] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.097636 9772 consensus_queue.cc:260] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.097637 9771 consensus_queue.cc:260] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.098349 9773 raft_consensus.cc:348] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.098615 9771 ts_tablet_manager.cc:1428] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:09:45.098621 9773 raft_consensus.cc:374] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.098671 9773 raft_consensus.cc:729] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.098785 9772 ts_tablet_manager.cc:1428] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:45.098975 9748 heartbeater.cc:503] Master 127.7.90.126:33287 was elected leader, sending a full tablet report...
I20260619 11:09:45.099144 9468 heartbeater.cc:503] Master 127.7.90.126:33287 was elected leader, sending a full tablet report...
I20260619 11:09:45.099083 9773 consensus_queue.cc:260] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
W20260619 11:09:45.099540 9469 tablet.cc:2369] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:45.099922 9773 ts_tablet_manager.cc:1428] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 11:09:45.100198 9608 heartbeater.cc:503] Master 127.7.90.126:33287 was elected leader, sending a full tablet report...
W20260619 11:09:45.268105 9749 tablet.cc:2369] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:09:45.314348 9609 tablet.cc:2369] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:45.322178 9777 raft_consensus.cc:482] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:45.322364 9777 raft_consensus.cc:504] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.323575 9777 leader_election.cc:288] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.328540 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.324267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:43441 (local address 127.7.90.67:46117)
0619 11:09:45.324382 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.324386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.324543 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:09:45.324711 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.324715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.324765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.324837 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.325094 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.325627 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.327357 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.327974 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.328077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.328139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.328349 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.328420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:45.328562 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.324164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:46117 (local address 127.7.90.65:43441)
0619 11:09:45.324469 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.324481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.324611 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.324866 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.324868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.324881 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.324989 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.324994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.325733 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.325737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.327236 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.327243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.328094 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.328101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.328219 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.328240 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.328331 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.328401 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":142,"spinlock_wait_cycles":1408,"thread_start_us":71,"threads_started":1}
I20260619 11:09:45.329252 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9799c0b533406687b9d064469d6561" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb" is_pre_election: true
I20260619 11:09:45.329355 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.323891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:38207 (local address 127.7.90.66:43409)
0619 11:09:45.324033 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.324038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.324176 (+ 138us) server_negotiation.cc:407] Connection header received
0619 11:09:45.324414 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.324417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.324476 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.324553 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.324863 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.325674 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.328214 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.328810 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.328907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.328961 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.329168 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.329245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:09:45.329608 9702 raft_consensus.cc:2454] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 0.
I20260619 11:09:45.329758 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.323816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.7.90.65:38207)
0619 11:09:45.324097 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.324112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.324262 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.324577 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.324580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.324614 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.324736 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.324742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.325826 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.325828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.327233 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.327243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.328922 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.328926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.329023 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.329049 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.329491 (+ 442us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.329610 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":79,"spinlock_wait_cycles":12032,"thread_start_us":67,"threads_started":1}
I20260619 11:09:45.330104 9354 leader_election.cc:302] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, 20d976bf731741c1a36632b5e16e83cb; no voters:
I20260619 11:09:45.330336 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9799c0b533406687b9d064469d6561" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979" is_pre_election: true
I20260619 11:09:45.330391 9777 raft_consensus.cc:2781] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:45.330504 9777 raft_consensus.cc:482] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:45.330574 9777 raft_consensus.cc:3037] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.330648 9562 raft_consensus.cc:2454] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 0.
I20260619 11:09:45.331799 9777 raft_consensus.cc:504] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.332158 9777 leader_election.cc:288] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 election: Requested vote from peers af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.332443 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9799c0b533406687b9d064469d6561" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979"
I20260619 11:09:45.332486 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9799c0b533406687b9d064469d6561" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb"
I20260619 11:09:45.332561 9562 raft_consensus.cc:3037] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.332581 9702 raft_consensus.cc:3037] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.333873 9562 raft_consensus.cc:2454] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 1.
I20260619 11:09:45.334174 9354 leader_election.cc:302] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:45.334261 9702 raft_consensus.cc:2454] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 1.
I20260619 11:09:45.334352 9777 raft_consensus.cc:2781] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:45.334530 9777 raft_consensus.cc:686] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 1 LEADER]: Becoming Leader. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Running, Role: LEADER
I20260619 11:09:45.334844 9777 consensus_queue.cc:237] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } }
I20260619 11:09:45.337848 9263 catalog_manager.cc:5495] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 174dc4d25230493391975a9db90a88d2 (127.7.90.65). New cstate: current_term: 1 leader_uuid: "174dc4d25230493391975a9db90a88d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } health_report { overall_health: HEALTHY } } }
Jun 19 11:09:45 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867384, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:45.383227 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.375241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33287 (local address 127.0.0.1:60662)
0619 11:09:45.375511 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.375521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.375638 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.375852 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.375854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.376285 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:45.376388 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.376394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.377201 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.377204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.377847 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.377856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.378320 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.378328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.378405 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.378908 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:45.378918 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:45.380264 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:45.381924 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.381938 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.381942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.382161 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.382382 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:45.382384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:45.382385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:45.382457 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:45.382755 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:45.382759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:45.382849 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.382943 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.383048 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260619 11:09:45.383244 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.375277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.7.90.126:33287)
0619 11:09:45.375410 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.375415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.375587 (+ 172us) server_negotiation.cc:407] Connection header received
0619 11:09:45.375720 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.375723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.375761 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.375837 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:45.376508 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.377022 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.377942 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.378180 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.378276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.380385 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:45.380388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:45.380391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:45.380411 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:45.381839 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.382239 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.382241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.382242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.382310 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:45.382525 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:45.382527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:45.382528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:45.382673 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:45.382743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.382992 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.383099 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":46}
I20260619 11:09:45.386823 9263 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:60662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:09:45.387521 9263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:45.392025 9542 tablet_service.cc:1467] Processing CreateTablet for tablet 33eb3fe7f8c0464495d34beb3ca7734f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fb5755259d748829b8b6313f7ec1e03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:45.392105 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 33eb3fe7f8c0464495d34beb3ca7734f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fb5755259d748829b8b6313f7ec1e03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:45.392017 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 33eb3fe7f8c0464495d34beb3ca7734f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4fb5755259d748829b8b6313f7ec1e03]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:09:45.392760 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33eb3fe7f8c0464495d34beb3ca7734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.392760 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33eb3fe7f8c0464495d34beb3ca7734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.392769 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33eb3fe7f8c0464495d34beb3ca7734f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.395252 9771 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: Bootstrap starting.
I20260619 11:09:45.395591 9772 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: Bootstrap starting.
I20260619 11:09:45.395679 9773 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: Bootstrap starting.
I20260619 11:09:45.396260 9771 tablet_bootstrap.cc:654] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.396595 9773 tablet_bootstrap.cc:654] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.396936 9772 tablet_bootstrap.cc:654] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.397804 9773 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: No bootstrap required, opened a new log
I20260619 11:09:45.397886 9773 ts_tablet_manager.cc:1397] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 11:09:45.398159 9772 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: No bootstrap required, opened a new log
I20260619 11:09:45.398231 9772 ts_tablet_manager.cc:1397] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:09:45.398376 9771 tablet_bootstrap.cc:492] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: No bootstrap required, opened a new log
I20260619 11:09:45.398340 9773 raft_consensus.cc:348] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.398442 9771 ts_tablet_manager.cc:1397] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:09:45.398437 9773 raft_consensus.cc:374] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.398484 9773 raft_consensus.cc:729] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.398617 9773 consensus_queue.cc:260] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.398604 9772 raft_consensus.cc:348] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.398700 9772 raft_consensus.cc:374] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.398744 9772 raft_consensus.cc:729] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.398844 9772 consensus_queue.cc:260] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.398851 9771 raft_consensus.cc:348] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.398869 9779 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.398974 9771 raft_consensus.cc:374] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.399005 9779 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.399013 9771 raft_consensus.cc:729] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.398856 9773 ts_tablet_manager.cc:1428] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:09:45.399123 9772 ts_tablet_manager.cc:1428] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:45.399153 9771 consensus_queue.cc:260] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.399457 9771 ts_tablet_manager.cc:1428] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:45.399435 9777 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.399422 9778 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.399539 9777 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.399549 9778 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.610549 9779 raft_consensus.cc:482] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:45.610671 9779 raft_consensus.cc:504] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.611701 9779 leader_election.cc:288] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.615394 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.611998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:50785 (local address 127.7.90.67:46117)
0619 11:09:45.612128 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.612133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.612350 (+ 217us) server_negotiation.cc:407] Connection header received
0619 11:09:45.612519 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.612521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.612560 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.612639 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.612912 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.613382 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.614455 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.614856 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.614988 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.615048 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.615232 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.615296 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:45.615419 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.611918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:46117 (local address 127.7.90.66:50785)
0619 11:09:45.612231 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.612241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.612389 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.612649 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.612652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.612667 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.612804 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.612810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.613485 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.613488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.614332 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.614340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.615015 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.615019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.615119 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.615136 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.615217 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.615278 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260619 11:09:45.615736 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.611918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:38439 (local address 127.7.90.66:50967)
0619 11:09:45.612337 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.612349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.612463 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.612702 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.612705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.612719 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:45.613169 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.613175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.613971 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.613975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.614694 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.614702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.615304 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.615307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.615403 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.615418 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.615528 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.615611 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":64,"thread_start_us":52,"threads_started":1}
I20260619 11:09:45.615793 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33eb3fe7f8c0464495d34beb3ca7734f" candidate_uuid: "af0803920cb44a72a2186fcf7ef67979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb" is_pre_election: true
I20260619 11:09:45.615859 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.611998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:50967 (local address 127.7.90.65:38439)
0619 11:09:45.612267 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.612281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.612411 (+ 130us) server_negotiation.cc:407] Connection header received
0619 11:09:45.612557 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.612560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.612604 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.612685 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:45.613276 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.613849 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.614822 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.615197 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.615301 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.615353 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.615671 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.615742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260619 11:09:45.615927 9702 raft_consensus.cc:2454] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0803920cb44a72a2186fcf7ef67979 in term 0.
I20260619 11:09:45.616237 9494 leader_election.cc:302] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d976bf731741c1a36632b5e16e83cb, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:45.616252 9422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33eb3fe7f8c0464495d34beb3ca7734f" candidate_uuid: "af0803920cb44a72a2186fcf7ef67979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2" is_pre_election: true
I20260619 11:09:45.616474 9422 raft_consensus.cc:2454] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0803920cb44a72a2186fcf7ef67979 in term 0.
I20260619 11:09:45.616514 9779 raft_consensus.cc:2781] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:45.616573 9779 raft_consensus.cc:482] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:45.616640 9779 raft_consensus.cc:3037] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.617792 9779 raft_consensus.cc:504] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.618120 9779 leader_election.cc:288] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [CANDIDATE]: Term 1 election: Requested vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.618459 9422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33eb3fe7f8c0464495d34beb3ca7734f" candidate_uuid: "af0803920cb44a72a2186fcf7ef67979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2"
I20260619 11:09:45.618603 9422 raft_consensus.cc:3037] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.618605 9702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33eb3fe7f8c0464495d34beb3ca7734f" candidate_uuid: "af0803920cb44a72a2186fcf7ef67979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb"
I20260619 11:09:45.618710 9702 raft_consensus.cc:3037] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.619756 9422 raft_consensus.cc:2454] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0803920cb44a72a2186fcf7ef67979 in term 1.
I20260619 11:09:45.619838 9702 raft_consensus.cc:2454] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0803920cb44a72a2186fcf7ef67979 in term 1.
I20260619 11:09:45.620036 9495 leader_election.cc:302] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:45.620186 9779 raft_consensus.cc:2781] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:45.620337 9779 raft_consensus.cc:686] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [term 1 LEADER]: Becoming Leader. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Running, Role: LEADER
I20260619 11:09:45.620645 9779 consensus_queue.cc:237] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.622826 9796 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: TxnStatusTablet state changed. Reason: New leader af0803920cb44a72a2186fcf7ef67979. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.622958 9796 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:45.623240 9798 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:09:45.623440 9798 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:09:45.623728 9264 catalog_manager.cc:5495] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 reported cstate change: term changed from 0 to 1, leader changed from <none> to af0803920cb44a72a2186fcf7ef67979 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } health_report { overall_health: UNKNOWN } } }
I20260619 11:09:45.692387 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.688657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.7.90.66:43409)
0619 11:09:45.688790 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.688794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.688808 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:09:45.688962 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.688976 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.689028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.689127 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:45.689357 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.689911 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.690782 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.691005 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.691127 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.691448 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.691928 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.692027 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.692167 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.692238 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:45.692374 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.688534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.0.0.1:40832)
0619 11:09:45.688679 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.688696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.688802 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.689142 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.689145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.689154 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:45.689255 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.689260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.690018 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.690030 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.690662 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.690670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.691146 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.691150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.691249 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.691300 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.692039 (+ 739us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.692054 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.692124 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.692208 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:09:45.697292 9422 raft_consensus.cc:1261] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Refusing update from remote peer af0803920cb44a72a2186fcf7ef67979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:45.697306 9702 raft_consensus.cc:1261] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Refusing update from remote peer af0803920cb44a72a2186fcf7ef67979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:45.697927 9779 consensus_queue.cc:1035] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:45.698083 9796 consensus_queue.cc:1035] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:45.700891 9777 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: TxnStatusTablet state changed. Reason: New leader af0803920cb44a72a2186fcf7ef67979. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.701010 9777 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.702999 9796 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.703101 9796 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:45.704157 9778 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: TxnStatusTablet state changed. Reason: New leader af0803920cb44a72a2186fcf7ef67979. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.704260 9778 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.704416 9777 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.704541 9777 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 174dc4d25230493391975a9db90a88d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.704909 9801 mvcc.cc:205] Tried to move back new op lower bound from 7298528811809509376 to 7298528811504619520. Current Snapshot: MvccSnapshot[applied={T|T < 7298528811809509376 or (T in {7298528811809509376})}]
I20260619 11:09:45.709606 9779 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.709704 9779 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P af0803920cb44a72a2186fcf7ef67979: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:09:45.710773 9778 tablet_replica.cc:439] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af0803920cb44a72a2186fcf7ef67979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } } }
I20260619 11:09:45.710879 9778 tablet_replica.cc:442] T 33eb3fe7f8c0464495d34beb3ca7734f P 20d976bf731741c1a36632b5e16e83cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:09:45.711295 9804 mvcc.cc:205] Tried to move back new op lower bound from 7298528811809509376 to 7298528811504619520. Current Snapshot: MvccSnapshot[applied={T|T < 7298528811809509376}]
I20260619 11:09:45.715421 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.710937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.7.90.65:38439)
0619 11:09:45.711223 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.711228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.711293 (+ 65us) server_negotiation.cc:407] Connection header received
0619 11:09:45.711561 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.711565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.711620 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.711685 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:45.712048 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.712788 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.714057 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.714281 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.714362 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.714681 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.715032 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.715111 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.715229 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.715273 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260619 11:09:45.725603 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.722190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:44817 (local address 127.7.90.66:43409)
0619 11:09:45.722304 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.722308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.722405 (+ 97us) server_negotiation.cc:407] Connection header received
0619 11:09:45.722528 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.722530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.722586 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.722647 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:45.722913 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.723378 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.724617 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.724804 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.724873 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.725191 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.725247 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.725293 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.725418 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.725455 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:09:45.725667 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.722050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.7.90.65:44817)
0619 11:09:45.722321 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.722335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.722440 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.722676 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.722680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.722690 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:45.722811 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.722820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.723478 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.723483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.724509 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.724521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.724900 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.724903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.724995 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.725040 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.725326 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.725333 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.725405 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.725488 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260619 11:09:45.732651 9702 raft_consensus.cc:1261] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Refusing update from remote peer 174dc4d25230493391975a9db90a88d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:45.732753 9562 raft_consensus.cc:1261] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Refusing update from remote peer 174dc4d25230493391975a9db90a88d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:45.733093 9777 consensus_queue.cc:1035] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:45.733266 9782 consensus_queue.cc:1035] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:45.735935 9802 mvcc.cc:205] Tried to move back new op lower bound from 7298528811956690944 to 7298528810334007296. Current Snapshot: MvccSnapshot[applied={T|T < 7298528811956690944}]
I20260619 11:09:45.755578 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.751543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:38439 (local address 127.7.90.66:52543)
0619 11:09:45.751866 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:45.751884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:45.751992 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:45.752242 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:45.752245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:45.752253 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:45.752368 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.752375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.753489 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.753494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.754169 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:45.754178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.754805 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.754809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:45.754904 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.754944 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.755252 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.755259 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:09:45.755330 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:09:45.755442 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
I20260619 11:09:45.755988 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:45.751662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:52543 (local address 127.7.90.65:38439)
0619 11:09:45.751765 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:45.751770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:45.751956 (+ 186us) server_negotiation.cc:407] Connection header received
0619 11:09:45.752081 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:45.752084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:45.752134 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:45.752221 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:45.752493 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.753312 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.754287 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:45.754630 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:45.754744 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:45.755080 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:45.755157 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:45.755236 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:45.755806 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 11:09:45.755870 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:09:45.778242 9264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:45.778439 9264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:45.780793 9264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Sending DeleteTablet for 3 replicas of tablet dd9799c0b533406687b9d064469d6561
I20260619 11:09:45.781446 9542 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9799c0b533406687b9d064469d6561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:45 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40830
I20260619 11:09:45.781517 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9799c0b533406687b9d064469d6561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:45 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:46176
I20260619 11:09:45.781801 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9799c0b533406687b9d064469d6561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:45 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47926
I20260619 11:09:45.781970 9822 tablet_replica.cc:330] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: stopping tablet replica
I20260619 11:09:45.782241 9822 raft_consensus.cc:2229] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:45.782424 9822 raft_consensus.cc:2258] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:45.782706 9821 tablet_replica.cc:330] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: stopping tablet replica
I20260619 11:09:45.783124 9264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:45.783501 9264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:45.782951 9821 raft_consensus.cc:2229] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:45.784204 9821 raft_consensus.cc:2258] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:45.786675 9821 ts_tablet_manager.cc:1905] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:45.788349 9402 tablet_service.cc:1467] Processing CreateTablet for tablet c45e03813eec480c98f26dff2c3a315a (DEFAULT_TABLE table=test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.788581 9823 tablet_replica.cc:330] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: stopping tablet replica
I20260619 11:09:45.788645 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45e03813eec480c98f26dff2c3a315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.788713 9682 tablet_service.cc:1467] Processing CreateTablet for tablet c45e03813eec480c98f26dff2c3a315a (DEFAULT_TABLE table=test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.788866 9823 raft_consensus.cc:2229] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:45.788991 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45e03813eec480c98f26dff2c3a315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.789209 9823 raft_consensus.cc:2258] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:45.790166 9821 ts_tablet_manager.cc:1918] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:45.790266 9821 log.cc:1196] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/dd9799c0b533406687b9d064469d6561
I20260619 11:09:45.790454 9823 ts_tablet_manager.cc:1905] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:45.790618 9821 ts_tablet_manager.cc:1939] T dd9799c0b533406687b9d064469d6561 P af0803920cb44a72a2186fcf7ef67979: Deleting consensus metadata
I20260619 11:09:45.791319 9772 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Bootstrap starting.
I20260619 11:09:45.791558 9771 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Bootstrap starting.
I20260619 11:09:45.791749 9822 ts_tablet_manager.cc:1905] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:45.791754 9542 tablet_service.cc:1467] Processing CreateTablet for tablet c45e03813eec480c98f26dff2c3a315a (DEFAULT_TABLE table=test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:45.791985 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45e03813eec480c98f26dff2c3a315a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:45.792296 9772 tablet_bootstrap.cc:654] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.792510 9771 tablet_bootstrap.cc:654] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.794508 9773 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Bootstrap starting.
I20260619 11:09:45.794574 9822 ts_tablet_manager.cc:1918] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:45.794667 9822 log.cc:1196] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/dd9799c0b533406687b9d064469d6561
I20260619 11:09:45.795099 9822 ts_tablet_manager.cc:1939] T dd9799c0b533406687b9d064469d6561 P 20d976bf731741c1a36632b5e16e83cb: Deleting consensus metadata
I20260619 11:09:45.795508 9773 tablet_bootstrap.cc:654] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:45.795809 9771 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: No bootstrap required, opened a new log
I20260619 11:09:45.795920 9771 ts_tablet_manager.cc:1397] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 11:09:45.796262 9823 ts_tablet_manager.cc:1918] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:45.796247 9250 catalog_manager.cc:4841] TS af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409): tablet dd9799c0b533406687b9d064469d6561 (table test-table [id=3910dbe438784967bd5c10e435345c51]) successfully deleted
I20260619 11:09:45.796473 9823 log.cc:1196] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/dd9799c0b533406687b9d064469d6561
I20260619 11:09:45.796425 9771 raft_consensus.cc:348] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.796548 9771 raft_consensus.cc:374] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.796587 9771 raft_consensus.cc:729] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.796729 9771 consensus_queue.cc:260] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.797055 9771 ts_tablet_manager.cc:1428] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:45.797092 9823 ts_tablet_manager.cc:1939] T dd9799c0b533406687b9d064469d6561 P 174dc4d25230493391975a9db90a88d2: Deleting consensus metadata
I20260619 11:09:45.797096 9773 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: No bootstrap required, opened a new log
I20260619 11:09:45.797212 9773 ts_tablet_manager.cc:1397] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:09:45.797374 9250 catalog_manager.cc:4841] TS 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117): tablet dd9799c0b533406687b9d064469d6561 (table test-table [id=3910dbe438784967bd5c10e435345c51]) successfully deleted
I20260619 11:09:45.797924 9773 raft_consensus.cc:348] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.798055 9773 raft_consensus.cc:374] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.798092 9773 raft_consensus.cc:729] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.798136 9251 catalog_manager.cc:4841] TS 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439): tablet dd9799c0b533406687b9d064469d6561 (table test-table [id=3910dbe438784967bd5c10e435345c51]) successfully deleted
I20260619 11:09:45.798239 9773 consensus_queue.cc:260] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.798616 9773 ts_tablet_manager.cc:1428] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 11:09:45.798914 9772 tablet_bootstrap.cc:492] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: No bootstrap required, opened a new log
I20260619 11:09:45.798991 9772 ts_tablet_manager.cc:1397] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 11:09:45.799417 9772 raft_consensus.cc:348] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.799530 9772 raft_consensus.cc:374] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:45.799563 9772 raft_consensus.cc:729] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:45.799700 9772 consensus_queue.cc:260] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.799989 9772 ts_tablet_manager.cc:1428] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:45.951215 9782 raft_consensus.cc:482] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:45.951346 9782 raft_consensus.cc:504] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.951817 9782 leader_election.cc:288] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.952297 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45e03813eec480c98f26dff2c3a315a" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979" is_pre_election: true
I20260619 11:09:45.952316 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45e03813eec480c98f26dff2c3a315a" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb" is_pre_election: true
I20260619 11:09:45.952472 9701 raft_consensus.cc:2454] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 0.
I20260619 11:09:45.952471 9562 raft_consensus.cc:2454] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 0.
I20260619 11:09:45.952773 9354 leader_election.cc:302] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, 20d976bf731741c1a36632b5e16e83cb; no voters:
I20260619 11:09:45.952947 9782 raft_consensus.cc:2781] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:45.953007 9782 raft_consensus.cc:482] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:45.953042 9782 raft_consensus.cc:3037] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.954160 9782 raft_consensus.cc:504] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.954483 9782 leader_election.cc:288] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 election: Requested vote from peers af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409), 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117)
I20260619 11:09:45.954869 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45e03813eec480c98f26dff2c3a315a" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979"
I20260619 11:09:45.954864 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45e03813eec480c98f26dff2c3a315a" candidate_uuid: "174dc4d25230493391975a9db90a88d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d976bf731741c1a36632b5e16e83cb"
I20260619 11:09:45.955006 9562 raft_consensus.cc:3037] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.955093 9701 raft_consensus.cc:3037] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:45.956015 9701 raft_consensus.cc:2454] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 1.
I20260619 11:09:45.956034 9562 raft_consensus.cc:2454] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174dc4d25230493391975a9db90a88d2 in term 1.
I20260619 11:09:45.956315 9354 leader_election.cc:302] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:45.956473 9782 raft_consensus.cc:2781] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:45.956562 9782 raft_consensus.cc:686] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 1 LEADER]: Becoming Leader. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Running, Role: LEADER
I20260619 11:09:45.956722 9782 consensus_queue.cc:237] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:45.958500 9265 catalog_manager.cc:5495] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 174dc4d25230493391975a9db90a88d2 (127.7.90.65). New cstate: current_term: 1 leader_uuid: "174dc4d25230493391975a9db90a88d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } health_report { overall_health: UNKNOWN } } }
I20260619 11:09:46.001134 9701 raft_consensus.cc:1261] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Refusing update from remote peer 174dc4d25230493391975a9db90a88d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:46.001240 9562 raft_consensus.cc:1261] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Refusing update from remote peer 174dc4d25230493391975a9db90a88d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:46.001507 9782 consensus_queue.cc:1035] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:46.001700 9777 consensus_queue.cc:1035] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:09:46.013547 9802 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:09:47.580459 9265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:47.580615 9265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:47.583359 9265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Sending DeleteTablet for 3 replicas of tablet c45e03813eec480c98f26dff2c3a315a
I20260619 11:09:47.583951 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet c45e03813eec480c98f26dff2c3a315a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47926
I20260619 11:09:47.584010 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet c45e03813eec480c98f26dff2c3a315a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:46176
I20260619 11:09:47.583988 9542 tablet_service.cc:1514] Processing DeleteTablet for tablet c45e03813eec480c98f26dff2c3a315a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40830
I20260619 11:09:47.584344 9855 tablet_replica.cc:330] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: stopping tablet replica
I20260619 11:09:47.584363 9856 tablet_replica.cc:330] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: stopping tablet replica
I20260619 11:09:47.584496 9855 raft_consensus.cc:2229] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:47.584501 9856 raft_consensus.cc:2229] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:47.584619 9856 raft_consensus.cc:2258] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.584637 9855 raft_consensus.cc:2258] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.584865 9854 tablet_replica.cc:330] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: stopping tablet replica
I20260619 11:09:47.585016 9854 raft_consensus.cc:2229] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:47.585239 9854 raft_consensus.cc:2258] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.585323 9855 ts_tablet_manager.cc:1905] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.585632 9856 ts_tablet_manager.cc:1905] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.586094 9854 ts_tablet_manager.cc:1905] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.588204 9855 ts_tablet_manager.cc:1918] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:09:47.588287 9855 log.cc:1196] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/c45e03813eec480c98f26dff2c3a315a
I20260619 11:09:47.588586 9855 ts_tablet_manager.cc:1939] T c45e03813eec480c98f26dff2c3a315a P 20d976bf731741c1a36632b5e16e83cb: Deleting consensus metadata
I20260619 11:09:47.588958 9856 ts_tablet_manager.cc:1918] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:09:47.589038 9856 log.cc:1196] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/c45e03813eec480c98f26dff2c3a315a
I20260619 11:09:47.589291 9856 ts_tablet_manager.cc:1939] T c45e03813eec480c98f26dff2c3a315a P af0803920cb44a72a2186fcf7ef67979: Deleting consensus metadata
I20260619 11:09:47.589385 9250 catalog_manager.cc:4841] TS 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117): tablet c45e03813eec480c98f26dff2c3a315a (table test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]) successfully deleted
I20260619 11:09:47.589984 9250 catalog_manager.cc:4841] TS af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409): tablet c45e03813eec480c98f26dff2c3a315a (table test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]) successfully deleted
I20260619 11:09:47.590350 9854 ts_tablet_manager.cc:1918] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:09:47.590444 9854 log.cc:1196] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/c45e03813eec480c98f26dff2c3a315a
I20260619 11:09:47.590714 9854 ts_tablet_manager.cc:1939] T c45e03813eec480c98f26dff2c3a315a P 174dc4d25230493391975a9db90a88d2: Deleting consensus metadata
I20260619 11:09:47.591307 9251 catalog_manager.cc:4841] TS 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439): tablet c45e03813eec480c98f26dff2c3a315a (table test-table [id=7044ed2098c246bc9c0953d7f5a3c3f1]) successfully deleted
Jun 19 11:09:47 dist-test-slave-f1t5 krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:09:47 dist-test-slave-f1t5 krb5kdc[9220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:47.619180 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.606140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.7.90.126:33287)
0619 11:09:47.606454 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.606460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.606476 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:09:47.606618 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.606621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.606668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.606740 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:47.607568 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.608111 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.609313 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.609590 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.609669 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.615778 (+ 6109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:47.615782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:47.615784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:47.615815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:47.617414 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:47.617862 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:47.617865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:47.617866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:47.617945 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:47.618304 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:47.618308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:47.618310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:47.618507 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:47.618612 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.618837 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.618942 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260619 11:09:47.622017 9265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:47.622488 9265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:47.626844 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 (DEFAULT_TABLE table=test-table [id=7d59080f90f147a283878b55453bffcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.626987 9542 tablet_service.cc:1467] Processing CreateTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 (DEFAULT_TABLE table=test-table [id=7d59080f90f147a283878b55453bffcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.627035 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 (DEFAULT_TABLE table=test-table [id=7d59080f90f147a283878b55453bffcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.627152 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999ca2a819342dbbe2991b38ae2ea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.627271 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999ca2a819342dbbe2991b38ae2ea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.627279 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1999ca2a819342dbbe2991b38ae2ea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.629820 9867 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Bootstrap starting.
I20260619 11:09:47.629923 9865 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Bootstrap starting.
I20260619 11:09:47.630244 9866 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Bootstrap starting.
I20260619 11:09:47.630682 9865 tablet_bootstrap.cc:654] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.630682 9867 tablet_bootstrap.cc:654] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.631062 9866 tablet_bootstrap.cc:654] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.631908 9865 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: No bootstrap required, opened a new log
I20260619 11:09:47.632002 9865 ts_tablet_manager.cc:1397] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:47.632162 9867 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: No bootstrap required, opened a new log
I20260619 11:09:47.632249 9867 ts_tablet_manager.cc:1397] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:47.632303 9866 tablet_bootstrap.cc:492] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: No bootstrap required, opened a new log
I20260619 11:09:47.632371 9866 ts_tablet_manager.cc:1397] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:47.632519 9865 raft_consensus.cc:348] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.632629 9865 raft_consensus.cc:374] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.632668 9865 raft_consensus.cc:729] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.632658 9867 raft_consensus.cc:348] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.632761 9867 raft_consensus.cc:374] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.632795 9867 raft_consensus.cc:729] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.632761 9866 raft_consensus.cc:348] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.632855 9866 raft_consensus.cc:374] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.632887 9866 raft_consensus.cc:729] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.632833 9865 consensus_queue.cc:260] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.632925 9867 consensus_queue.cc:260] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.632997 9866 consensus_queue.cc:260] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.633224 9867 ts_tablet_manager.cc:1428] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:09:47.633349 9866 ts_tablet_manager.cc:1428] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:47.633869 9865 ts_tablet_manager.cc:1428] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:47.782685 9871 raft_consensus.cc:482] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:47.782845 9871 raft_consensus.cc:504] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.784015 9871 leader_election.cc:288] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409)
I20260619 11:09:47.788229 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.784415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:58949 (local address 127.7.90.65:38439)
0619 11:09:47.784775 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.784785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.784837 (+ 52us) server_negotiation.cc:407] Connection header received
0619 11:09:47.785023 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.785026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.785076 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.785170 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:47.785444 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.785992 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787158 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.787575 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787689 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.787756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.787992 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.788073 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260619 11:09:47.788229 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.784498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:46839 (local address 127.7.90.66:43409)
0619 11:09:47.784805 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.784815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.784960 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:09:47.785116 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.785119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.785165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.785260 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:47.785545 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.786076 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787178 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.787595 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787715 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.787780 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.787992 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.788078 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260619 11:09:47.788244 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.784288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:38439 (local address 127.7.90.67:58949)
0619 11:09:47.784752 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:47.784767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:47.784908 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:47.785178 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:47.785181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:47.785197 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:47.785328 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.785333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.786118 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.786125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.787034 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.787043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.787706 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.787825 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.787880 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 11:09:47.787976 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 11:09:47.788073 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":70,"thread_start_us":161,"threads_started":1}
I20260619 11:09:47.788247 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.784288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.7.90.67:46839)
0619 11:09:47.784866 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:47.784879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:47.785011 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:47.785271 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:47.785274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:47.785288 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:47.785414 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.785419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.786211 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.786214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.787048 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.787057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.787715 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.787719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.787868 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.787887 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:09:47.787976 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:09:47.788079 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":203,"thread_start_us":79,"threads_started":1}
I20260619 11:09:47.788939 9422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999ca2a819342dbbe2991b38ae2ea92" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2" is_pre_election: true
I20260619 11:09:47.788939 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999ca2a819342dbbe2991b38ae2ea92" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979" is_pre_election: true
I20260619 11:09:47.789136 9562 raft_consensus.cc:2454] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 0.
I20260619 11:09:47.789136 9422 raft_consensus.cc:2454] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 0.
I20260619 11:09:47.789609 9634 leader_election.cc:302] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, 20d976bf731741c1a36632b5e16e83cb, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:47.789884 9871 raft_consensus.cc:2781] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:47.789970 9871 raft_consensus.cc:482] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:47.790033 9871 raft_consensus.cc:3037] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:47.791191 9871 raft_consensus.cc:504] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.791587 9871 leader_election.cc:288] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 election: Requested vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409)
I20260619 11:09:47.791994 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999ca2a819342dbbe2991b38ae2ea92" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979"
I20260619 11:09:47.792084 9422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1999ca2a819342dbbe2991b38ae2ea92" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2"
I20260619 11:09:47.792141 9562 raft_consensus.cc:3037] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:47.792227 9422 raft_consensus.cc:3037] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:47.793217 9562 raft_consensus.cc:2454] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 1.
I20260619 11:09:47.793335 9422 raft_consensus.cc:2454] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 1.
I20260619 11:09:47.793496 9634 leader_election.cc:302] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d976bf731741c1a36632b5e16e83cb, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:47.793740 9871 raft_consensus.cc:2781] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:47.793975 9871 raft_consensus.cc:686] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 1 LEADER]: Becoming Leader. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Running, Role: LEADER
I20260619 11:09:47.794237 9871 consensus_queue.cc:237] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.796635 9264 catalog_manager.cc:5495] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d976bf731741c1a36632b5e16e83cb (127.7.90.67). New cstate: current_term: 1 leader_uuid: "20d976bf731741c1a36632b5e16e83cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } health_report { overall_health: HEALTHY } } }
I20260619 11:09:47.831195 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.827286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.7.90.67:46117)
0619 11:09:47.827591 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.827600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.827612 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:47.827690 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.827692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.827741 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.827869 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:47.828146 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.828704 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.829629 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.829880 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.829995 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.830276 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:47.830773 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:47.830909 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.830972 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.831027 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260619 11:09:47.841701 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.838047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.7.90.67:54013)
0619 11:09:47.838365 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:47.838393 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:47.838514 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:47.838770 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:47.838773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:47.838781 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:47.838906 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.838912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.839633 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.839636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.840445 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.840454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.840903 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.840906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.840999 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.841055 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:47.841373 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:47.841380 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:09:47.841455 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:09:47.841555 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260619 11:09:47.841689 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.838178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:54013 (local address 127.7.90.66:43409)
0619 11:09:47.838320 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.838327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.838463 (+ 136us) server_negotiation.cc:407] Connection header received
0619 11:09:47.838630 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.838634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.838679 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.838762 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:47.839016 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.839508 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.840563 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.840782 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.840913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.841213 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:47.841283 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:47.841357 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.841466 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.841542 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:09:47.847926 9422 raft_consensus.cc:1261] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Refusing update from remote peer 20d976bf731741c1a36632b5e16e83cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:47.848393 9871 consensus_queue.cc:1035] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:47.853592 9562 raft_consensus.cc:1261] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Refusing update from remote peer 20d976bf731741c1a36632b5e16e83cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:47.854377 9876 consensus_queue.cc:1035] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:47.867347 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.863890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:55259 (local address 127.7.90.67:46117)
0619 11:09:47.864000 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:47.864005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:47.864134 (+ 129us) server_negotiation.cc:407] Connection header received
0619 11:09:47.864259 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:47.864261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:47.864327 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:47.864404 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:47.864669 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.865158 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.866187 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.866425 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.866539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.866798 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:47.866878 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:47.866996 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:47.867146 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 11:09:47.867205 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:09:47.867359 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:47.863807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:46117 (local address 127.7.90.66:55259)
0619 11:09:47.864040 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:47.864057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:47.864170 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:47.864454 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:47.864457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:47.864466 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:09:47.864578 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.864583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.865241 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.865244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.866071 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:47.866081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:47.866543 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:47.866547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:47.866641 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:47.866672 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:09:47.867032 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:09:47.867041 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:09:47.867119 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:09:47.867220 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260619 11:09:47.890877 9265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:47.891083 9265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:47.893421 9265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Sending DeleteTablet for 3 replicas of tablet 1999ca2a819342dbbe2991b38ae2ea92
I20260619 11:09:47.894070 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47926
I20260619 11:09:47.894271 9854 tablet_replica.cc:330] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: stopping tablet replica
I20260619 11:09:47.894402 9854 raft_consensus.cc:2229] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:47.894563 9854 raft_consensus.cc:2258] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.894824 9542 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40830
I20260619 11:09:47.895017 9856 tablet_replica.cc:330] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: stopping tablet replica
I20260619 11:09:47.895133 9856 raft_consensus.cc:2229] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:47.895255 9856 raft_consensus.cc:2258] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.895370 9854 ts_tablet_manager.cc:1905] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.895460 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1999ca2a819342dbbe2991b38ae2ea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:47 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:46176
I20260619 11:09:47.895634 9855 tablet_replica.cc:330] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: stopping tablet replica
I20260619 11:09:47.895738 9855 raft_consensus.cc:2229] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:47.896054 9855 raft_consensus.cc:2258] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:47.896663 9855 ts_tablet_manager.cc:1905] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.897755 9265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:09:47.898109 9854 ts_tablet_manager.cc:1918] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:47.898180 9854 log.cc:1196] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/1999ca2a819342dbbe2991b38ae2ea92
W20260619 11:09:47.898221 9265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:47.898432 9854 ts_tablet_manager.cc:1939] T 1999ca2a819342dbbe2991b38ae2ea92 P 174dc4d25230493391975a9db90a88d2: Deleting consensus metadata
I20260619 11:09:47.899196 9251 catalog_manager.cc:4841] TS 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439): tablet 1999ca2a819342dbbe2991b38ae2ea92 (table test-table [id=7d59080f90f147a283878b55453bffcb]) successfully deleted
I20260619 11:09:47.899518 9855 ts_tablet_manager.cc:1918] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:47.899593 9855 log.cc:1196] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/1999ca2a819342dbbe2991b38ae2ea92
I20260619 11:09:47.899847 9855 ts_tablet_manager.cc:1939] T 1999ca2a819342dbbe2991b38ae2ea92 P 20d976bf731741c1a36632b5e16e83cb: Deleting consensus metadata
I20260619 11:09:47.900492 9250 catalog_manager.cc:4841] TS 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117): tablet 1999ca2a819342dbbe2991b38ae2ea92 (table test-table [id=7d59080f90f147a283878b55453bffcb]) successfully deleted
I20260619 11:09:47.900501 9856 ts_tablet_manager.cc:1905] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:47.903033 9856 ts_tablet_manager.cc:1918] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:09:47.903012 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 3d058d36921043e985f77f1ef49fb492 (DEFAULT_TABLE table=test-table [id=4817a9893e3844a29995bdf17f6d0644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.903105 9856 log.cc:1196] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/1999ca2a819342dbbe2991b38ae2ea92
I20260619 11:09:47.903276 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d058d36921043e985f77f1ef49fb492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.903482 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 3d058d36921043e985f77f1ef49fb492 (DEFAULT_TABLE table=test-table [id=4817a9893e3844a29995bdf17f6d0644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.903704 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d058d36921043e985f77f1ef49fb492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.905233 9867 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: Bootstrap starting.
I20260619 11:09:47.905963 9866 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Bootstrap starting.
I20260619 11:09:47.905990 9542 tablet_service.cc:1467] Processing CreateTablet for tablet 3d058d36921043e985f77f1ef49fb492 (DEFAULT_TABLE table=test-table [id=4817a9893e3844a29995bdf17f6d0644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:47.906086 9867 tablet_bootstrap.cc:654] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.906258 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d058d36921043e985f77f1ef49fb492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:47.906895 9866 tablet_bootstrap.cc:654] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.908267 9856 ts_tablet_manager.cc:1939] T 1999ca2a819342dbbe2991b38ae2ea92 P af0803920cb44a72a2186fcf7ef67979: Deleting consensus metadata
I20260619 11:09:47.908540 9866 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: No bootstrap required, opened a new log
I20260619 11:09:47.908617 9866 ts_tablet_manager.cc:1397] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:09:47.909000 9250 catalog_manager.cc:4841] TS af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409): tablet 1999ca2a819342dbbe2991b38ae2ea92 (table test-table [id=7d59080f90f147a283878b55453bffcb]) successfully deleted
I20260619 11:09:47.909056 9867 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: No bootstrap required, opened a new log
I20260619 11:09:47.909108 9865 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: Bootstrap starting.
I20260619 11:09:47.909133 9867 ts_tablet_manager.cc:1397] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 11:09:47.909065 9866 raft_consensus.cc:348] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.909176 9866 raft_consensus.cc:374] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.909207 9866 raft_consensus.cc:729] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.909325 9866 consensus_queue.cc:260] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.909555 9867 raft_consensus.cc:348] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.909626 9866 ts_tablet_manager.cc:1428] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:47.909662 9867 raft_consensus.cc:374] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.909688 9867 raft_consensus.cc:729] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174dc4d25230493391975a9db90a88d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.909830 9867 consensus_queue.cc:260] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.909968 9865 tablet_bootstrap.cc:654] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:47.910122 9867 ts_tablet_manager.cc:1428] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:09:47.911090 9865 tablet_bootstrap.cc:492] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: No bootstrap required, opened a new log
I20260619 11:09:47.911170 9865 ts_tablet_manager.cc:1397] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:47.911561 9865 raft_consensus.cc:348] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.911696 9865 raft_consensus.cc:374] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:47.911734 9865 raft_consensus.cc:729] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0803920cb44a72a2186fcf7ef67979, State: Initialized, Role: FOLLOWER
I20260619 11:09:47.911859 9865 consensus_queue.cc:260] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:47.912142 9865 ts_tablet_manager.cc:1428] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:09:48.161319 9871 raft_consensus.cc:482] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:48.161473 9871 raft_consensus.cc:504] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:48.161963 9871 leader_election.cc:288] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409)
I20260619 11:09:48.162503 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d058d36921043e985f77f1ef49fb492" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979" is_pre_election: true
I20260619 11:09:48.162504 9421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d058d36921043e985f77f1ef49fb492" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2" is_pre_election: true
I20260619 11:09:48.162662 9421 raft_consensus.cc:2454] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 0.
I20260619 11:09:48.162662 9562 raft_consensus.cc:2454] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 0.
I20260619 11:09:48.162985 9634 leader_election.cc:302] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d976bf731741c1a36632b5e16e83cb, af0803920cb44a72a2186fcf7ef67979; no voters:
I20260619 11:09:48.163174 9871 raft_consensus.cc:2781] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:48.163239 9871 raft_consensus.cc:482] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:48.163316 9871 raft_consensus.cc:3037] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:48.164294 9871 raft_consensus.cc:504] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:48.164646 9871 leader_election.cc:288] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 election: Requested vote from peers 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439), af0803920cb44a72a2186fcf7ef67979 (127.7.90.66:43409)
I20260619 11:09:48.164971 9421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d058d36921043e985f77f1ef49fb492" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174dc4d25230493391975a9db90a88d2"
I20260619 11:09:48.165001 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d058d36921043e985f77f1ef49fb492" candidate_uuid: "20d976bf731741c1a36632b5e16e83cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0803920cb44a72a2186fcf7ef67979"
I20260619 11:09:48.165092 9421 raft_consensus.cc:3037] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:48.165139 9562 raft_consensus.cc:3037] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:48.166101 9421 raft_consensus.cc:2454] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 1.
I20260619 11:09:48.166190 9562 raft_consensus.cc:2454] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d976bf731741c1a36632b5e16e83cb in term 1.
I20260619 11:09:48.166528 9635 leader_election.cc:302] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174dc4d25230493391975a9db90a88d2, 20d976bf731741c1a36632b5e16e83cb; no voters:
I20260619 11:09:48.166693 9871 raft_consensus.cc:2781] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:48.166785 9871 raft_consensus.cc:686] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 1 LEADER]: Becoming Leader. State: Replica: 20d976bf731741c1a36632b5e16e83cb, State: Running, Role: LEADER
I20260619 11:09:48.166944 9871 consensus_queue.cc:237] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } }
I20260619 11:09:48.168510 9264 catalog_manager.cc:5495] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d976bf731741c1a36632b5e16e83cb (127.7.90.67). New cstate: current_term: 1 leader_uuid: "20d976bf731741c1a36632b5e16e83cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d976bf731741c1a36632b5e16e83cb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 46117 } health_report { overall_health: HEALTHY } } }
I20260619 11:09:48.216045 9562 raft_consensus.cc:1261] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Refusing update from remote peer 20d976bf731741c1a36632b5e16e83cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:48.216084 9421 raft_consensus.cc:1261] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Refusing update from remote peer 20d976bf731741c1a36632b5e16e83cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:48.216447 9887 consensus_queue.cc:1035] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0803920cb44a72a2186fcf7ef67979" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:48.216646 9871 consensus_queue.cc:1035] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "174dc4d25230493391975a9db90a88d2" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:09:48.229307 9836 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:09:49.775444 9264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:49.775604 9264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:49.777983 9264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15dff3286224f0ca8828cb255cfe6c9: Sending DeleteTablet for 3 replicas of tablet 3d058d36921043e985f77f1ef49fb492
I20260619 11:09:49.778491 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d058d36921043e985f77f1ef49fb492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:49 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47926
I20260619 11:09:49.778563 9541 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d058d36921043e985f77f1ef49fb492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:49 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40830
I20260619 11:09:49.778640 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d058d36921043e985f77f1ef49fb492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:49 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:46176
I20260619 11:09:49.778867 9917 tablet_replica.cc:330] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2: stopping tablet replica
I20260619 11:09:49.778990 9917 raft_consensus.cc:2229] T 3d058d36921043e985f77f1ef49fb492 P 174dc4d25230493391975a9db90a88d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:49.779007 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 9337
I20260619 11:09:49.785715 9918 tablet_replica.cc:330] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: stopping tablet replica
I20260619 11:09:49.785874 9918 raft_consensus.cc:2229] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:49.786026 9918 raft_consensus.cc:2258] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:49.786262 9919 tablet_replica.cc:330] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: stopping tablet replica
I20260619 11:09:49.786417 9919 raft_consensus.cc:2229] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:49.786612 9919 raft_consensus.cc:2258] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:49.786810 9918 ts_tablet_manager.cc:1905] T 3d058d36921043e985f77f1ef49fb492 P af0803920cb44a72a2186fcf7ef67979: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:09:49.787257 9251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:38439 (error 108)
I20260619 11:09:49.787601 9919 ts_tablet_manager.cc:1905] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:49.788012 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 9477
I20260619 11:09:49.789695 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:49.787921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:38439 (local address 127.0.0.1:47950)
0619 11:09:49.789486 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:49.789592 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1475,"thread_start_us":81,"threads_started":1}
W20260619 11:09:49.789789 9251 catalog_manager.cc:4584] TS 174dc4d25230493391975a9db90a88d2 (127.7.90.65:38439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d058d36921043e985f77f1ef49fb492: Network error: Client connection negotiation failed: client connection to 127.7.90.65:38439: connect: Connection refused (error 111)
I20260619 11:09:49.790727 9919 ts_tablet_manager.cc:1918] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:09:49.790791 9919 log.cc:1196] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/3d058d36921043e985f77f1ef49fb492
I20260619 11:09:49.791095 9919 ts_tablet_manager.cc:1939] T 3d058d36921043e985f77f1ef49fb492 P 20d976bf731741c1a36632b5e16e83cb: Deleting consensus metadata
I20260619 11:09:49.791832 9250 catalog_manager.cc:4841] TS 20d976bf731741c1a36632b5e16e83cb (127.7.90.67:46117): tablet 3d058d36921043e985f77f1ef49fb492 (table test-table [id=4817a9893e3844a29995bdf17f6d0644]) successfully deleted
I20260619 11:09:49.794673 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 9617
I20260619 11:09:49.796094 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:49.794817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:43409 (local address 127.0.0.1:40848)
0619 11:09:49.795948 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:49.796009 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:43409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1011}
I20260619 11:09:49.802003 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 9233
2026-06-19T11:09:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6796 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:49 dist-test-slave-f1t5 krb5kdc[9924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:49 dist-test-slave-f1t5 krb5kdc[9924](info): set up 2 sockets
Jun 19 11:09:49 dist-test-slave-f1t5 krb5kdc[9924](info): commencing operation
krb5kdc: starting...
W20260619 11:09:50.825203 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:50 dist-test-slave-f1t5 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:50.942559 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:44261
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:51.051635 9937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:51.051877 9937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:51.051916 9937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:51.055733 9937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:51.055783 9937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:51.055804 9937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:51.055819 9937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:51.055835 9937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:51.060827 9937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:44261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:51.061777 9937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:51.062702 9937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:51.070851 9943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:51.070922 9937 server_base.cc:1034] running on GCE node
W20260619 11:09:51.070837 9945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.071026 9942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:51.071715 9937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:51.072731 9937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:51.073880 9937 hybrid_clock.cc:648] HybridClock initialized: now 1781867391073854 us; error 41 us; skew 500 ppm
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:51.076530 9937 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:51.077402 9937 webserver.cc:460] Webserver started at http://127.7.90.126:41711/ using document root <none> and password file <none>
I20260619 11:09:51.077970 9937 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:51.078032 9937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:51.078238 9937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:51.080178 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "6b664db730fe45578382d711d9ee3b14"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "186c090e88e92100bc6202a529da0c40"
server_key_iv: "b42afdcc2bb53d07453033c822d508cb"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.080713 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "6b664db730fe45578382d711d9ee3b14"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "186c090e88e92100bc6202a529da0c40"
server_key_iv: "b42afdcc2bb53d07453033c822d508cb"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.084208 9937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:09:51.086541 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.087736 9937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:51.087874 9937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "6b664db730fe45578382d711d9ee3b14"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "186c090e88e92100bc6202a529da0c40"
server_key_iv: "b42afdcc2bb53d07453033c822d508cb"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.087998 9937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:51.101665 9937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:51.104296 9937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:51.104483 9937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:51.111344 9937 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:44261
I20260619 11:09:51.111395 10004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:44261 every 8 connection(s)
I20260619 11:09:51.112300 9937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:51.115836 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:51.118016 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 9937
I20260619 11:09:51.118146 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:51.118414 7529 external_mini_cluster.cc:1428] Setting key 32462324a2c30b2a9648288f03f0266a
I20260619 11:09:51.122370 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14: Bootstrap starting.
I20260619 11:09:51.124744 10005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:51.125644 10005 log.cc:824] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:51.127684 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14: No bootstrap required, opened a new log
I20260619 11:09:51.130528 10005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } }
I20260619 11:09:51.130730 10005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:51.130797 10005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b664db730fe45578382d711d9ee3b14, State: Initialized, Role: FOLLOWER
I20260619 11:09:51.131266 10005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } }
I20260619 11:09:51.131382 10005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:51.131472 10005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:51.131582 10005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:51.132562 10005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } }
I20260619 11:09:51.132902 10005 leader_election.cc:302] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b664db730fe45578382d711d9ee3b14; no voters:
I20260619 11:09:51.133165 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.119789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.7.90.126:44261)
0619 11:09:51.120205 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:51.120219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:51.120253 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:09:51.120863 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:51.120880 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:51.121202 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:51.121526 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:51.122322 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.123322 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.124500 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.125008 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.125134 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.127884 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:51.127889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:51.127903 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:51.127933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:51.130091 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.130584 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.130589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.130593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.130707 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.131020 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.131026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.131030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.131472 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:51.131696 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:51.132002 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 11:09:51.132332 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":138,"threads_started":1}
I20260619 11:09:51.133213 10005 leader_election.cc:288] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:51.133313 10010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:51.133553 10010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [term 1 LEADER]: Becoming Leader. State: Replica: 6b664db730fe45578382d711d9ee3b14, State: Running, Role: LEADER
I20260619 11:09:51.133881 10010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } }
I20260619 11:09:51.134456 10005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:51.135799 10012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b664db730fe45578382d711d9ee3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } } }
I20260619 11:09:51.135917 10012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:51.135911 10011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b664db730fe45578382d711d9ee3b14. Latest consensus state: current_term: 1 leader_uuid: "6b664db730fe45578382d711d9ee3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b664db730fe45578382d711d9ee3b14" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44261 } } }
I20260619 11:09:51.136059 10011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:51.136391 10021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:51.139308 10021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:51.144676 10021 catalog_manager.cc:1347] Generated new cluster ID: fb59fcf511214e399235cb9fb2721376
I20260619 11:09:51.144897 10021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:51.159291 10021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:51.160188 10021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:51.166049 10021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b664db730fe45578382d711d9ee3b14: Generated new TSK 0
I20260619 11:09:51.166658 10021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:51.202169 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:51.308910 10031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:51.309144 10031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:51.309183 10031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:51.313060 10031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:51.313114 10031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:51.313207 10031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:51.318845 10031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:51.319798 10031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:51.320775 10031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:51.329941 10039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.329931 10037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.329942 10036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:51.330323 10031 server_base.cc:1034] running on GCE node
I20260619 11:09:51.330751 10031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:51.331323 10031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:51.332501 10031 hybrid_clock.cc:648] HybridClock initialized: now 1781867391332489 us; error 23 us; skew 500 ppm
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:51.335376 10031 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:51.336278 10031 webserver.cc:460] Webserver started at http://127.7.90.65:44283/ using document root <none> and password file <none>
I20260619 11:09:51.336814 10031 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:51.336875 10031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:51.337035 10031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:51.338960 10031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "5980722823ce4b8cb03f403b1892f6bf"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "c07d0b411ca99250e5afa9b744ca3bd8"
server_key_iv: "b219b2adbee1debd9764b1344c29ad7d"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.339447 10031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "5980722823ce4b8cb03f403b1892f6bf"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "c07d0b411ca99250e5afa9b744ca3bd8"
server_key_iv: "b219b2adbee1debd9764b1344c29ad7d"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.342833 10031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:51.345039 10046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.346246 10031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:51.346370 10031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "5980722823ce4b8cb03f403b1892f6bf"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "c07d0b411ca99250e5afa9b744ca3bd8"
server_key_iv: "b219b2adbee1debd9764b1344c29ad7d"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.346482 10031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:51.359325 10031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:51.361909 10031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:51.362092 10031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:51.362634 10031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:51.363356 10031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:51.363438 10031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.363524 10031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:51.363579 10031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.372036 10031 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36037
I20260619 11:09:51.372066 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36037 every 8 connection(s)
I20260619 11:09:51.372817 10031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:51.377720 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10031
I20260619 11:09:51.377831 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:51.378093 7529 external_mini_cluster.cc:1428] Setting key ea57216b3683b87acf85839d6ee011f2
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:51.386278 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.374529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:53811 (local address 127.7.90.126:44261)
0619 11:09:51.374658 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:51.374664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:51.375570 (+ 906us) server_negotiation.cc:407] Connection header received
0619 11:09:51.376440 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:51.376443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:51.376487 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:51.376574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:51.377703 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.378483 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.379659 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.380066 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.380186 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.382610 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:51.382613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:51.382617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:51.382659 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:51.384181 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.384677 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.384683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.384687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.384780 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.385126 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.385133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.385136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.385387 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:51.385501 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:51.385975 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 11:09:51.386110 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:09:51.386924 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.374872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:44261 (local address 127.7.90.65:53811)
0619 11:09:51.375458 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:51.375493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:51.376176 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:51.376708 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:51.376714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:51.377071 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:51.377504 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.377517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.378601 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.378607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.379541 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.379549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.380201 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.380204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.380329 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.380909 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:51.380923 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:51.382457 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:51.384280 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.384284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.384294 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.384571 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.384880 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.384882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.384884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.385027 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.385492 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:51.385496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:51.385746 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 11:09:51.385891 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 11:09:51.386111 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:09:51.388046 10160 heartbeater.cc:347] Connected to a master server at 127.7.90.126:44261
I20260619 11:09:51.388283 10160 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:51.388731 10160 heartbeater.cc:511] Master 127.7.90.126:44261 requested a full tablet report, sending...
I20260619 11:09:51.390219 9969 ts_manager.cc:194] Registered new tserver with Master: 5980722823ce4b8cb03f403b1892f6bf (127.7.90.65:36037)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:09:51.391459 9969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:53811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:51.404702 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:51.516137 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:51.516381 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:51.516420 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:51.520274 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:51.520320 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:51.520390 10165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:51.525774 10165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:51.526816 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:51.527761 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:51.536880 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.536899 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.536886 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:51.537202 10165 server_base.cc:1034] running on GCE node
I20260619 11:09:51.537725 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:51.538303 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:51.539460 10165 hybrid_clock.cc:648] HybridClock initialized: now 1781867391539434 us; error 35 us; skew 500 ppm
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:51.542263 10165 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:51.543048 10165 webserver.cc:460] Webserver started at http://127.7.90.66:36065/ using document root <none> and password file <none>
I20260619 11:09:51.543530 10165 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:51.543587 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:51.543735 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:51.545804 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "ae15427450bd46b9afe3d2fb8b27a9a1"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "ae2f5290a1fdf8845b736d61f6d21e6c"
server_key_iv: "5d669457eca80929f8e87e30299cd40f"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.546378 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "ae15427450bd46b9afe3d2fb8b27a9a1"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "ae2f5290a1fdf8845b736d61f6d21e6c"
server_key_iv: "5d669457eca80929f8e87e30299cd40f"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.550225 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 11:09:51.552446 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.553728 10165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:51.553848 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "ae15427450bd46b9afe3d2fb8b27a9a1"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "ae2f5290a1fdf8845b736d61f6d21e6c"
server_key_iv: "5d669457eca80929f8e87e30299cd40f"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.553952 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:51.571230 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:51.574023 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:51.574237 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:51.574843 10165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:51.575632 10165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:51.575681 10165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.575752 10165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:51.575799 10165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.584702 10165 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:40107
I20260619 11:09:51.584720 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:40107 every 8 connection(s)
I20260619 11:09:51.585609 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:51.593429 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10165
I20260619 11:09:51.593570 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:51.593914 7529 external_mini_cluster.cc:1428] Setting key 840578ba8bd7d2ae7159474bdcf83446
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:51.600586 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.587283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:58363 (local address 127.7.90.126:44261)
0619 11:09:51.587431 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:51.587434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:51.588360 (+ 926us) server_negotiation.cc:407] Connection header received
0619 11:09:51.589238 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:51.589242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:51.589301 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:51.589420 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:51.590523 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.591120 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.592622 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.592855 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.592943 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.595528 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:51.595531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:51.595532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:51.595557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:51.598393 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.599004 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.599007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.599008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.599091 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.599475 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.599477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.599478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.599631 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:51.599752 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:51.600374 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 11:09:51.600472 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:09:51.601816 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.587591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:44261 (local address 127.7.90.66:58363)
0619 11:09:51.588204 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:51.588243 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:51.589036 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:51.589602 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:51.589610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:51.589945 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:51.590373 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.590387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.591268 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.591273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.592482 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.592492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.592990 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.592993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.593129 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.593788 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:51.593806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:51.595367 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:51.598506 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.598515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.598531 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.598887 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.599185 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.599190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.599195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.599367 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.599769 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:51.599778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:51.600112 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 11:09:51.600327 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 11:09:51.600698 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":101,"threads_started":1}
I20260619 11:09:51.602895 10294 heartbeater.cc:347] Connected to a master server at 127.7.90.126:44261
I20260619 11:09:51.603111 10294 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:51.603549 10294 heartbeater.cc:511] Master 127.7.90.126:44261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:09:51.604542 9969 ts_manager.cc:194] Registered new tserver with Master: ae15427450bd46b9afe3d2fb8b27a9a1 (127.7.90.66:40107)
I20260619 11:09:51.605083 9969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:58363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:51.620594 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:51.725728 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:51.726007 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:51.726084 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:51.730051 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:51.730137 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:51.730244 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:51.735396 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:51.736402 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:51.737336 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:09:51.746270 10299 server_base.cc:1034] running on GCE node
W20260619 11:09:51.746301 10304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.746250 10305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:51.746254 10307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:51.746847 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:51.747411 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:51.748588 10299 hybrid_clock.cc:648] HybridClock initialized: now 1781867391748576 us; error 24 us; skew 500 ppm
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:51.751132 10299 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:51.751855 10299 webserver.cc:460] Webserver started at http://127.7.90.67:40001/ using document root <none> and password file <none>
I20260619 11:09:51.752331 10299 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:51.752410 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:51.752625 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:51.754509 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "21b33bdccb38469bb4557203f91d006c"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "8ad7716f28178ddc8ef60dc4b70facce"
server_key_iv: "411485acefe183170d4592470abc8efe"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.755044 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "21b33bdccb38469bb4557203f91d006c"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "8ad7716f28178ddc8ef60dc4b70facce"
server_key_iv: "411485acefe183170d4592470abc8efe"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.758556 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:51.760757 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.761976 10299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:51.762125 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "21b33bdccb38469bb4557203f91d006c"
format_stamp: "Formatted at 2026-06-19 11:09:51 on dist-test-slave-f1t5"
server_key: "8ad7716f28178ddc8ef60dc4b70facce"
server_key_iv: "411485acefe183170d4592470abc8efe"
server_key_version: "encryptionkey@0"
I20260619 11:09:51.762228 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:51.802848 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:51.806152 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:51.806361 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:51.806934 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:51.807751 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:51.807832 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.807922 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:51.807974 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:51.816711 10299 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44143
I20260619 11:09:51.816742 10427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44143 every 8 connection(s)
I20260619 11:09:51.817695 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[9924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:51.826546 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10299
I20260619 11:09:51.826661 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:51.826944 7529 external_mini_cluster.cc:1428] Setting key a0fd5b45023da7f6a4dc27ee9d2586e4
I20260619 11:09:51.830317 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.819414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:41509 (local address 127.7.90.126:44261)
0619 11:09:51.819546 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:51.819549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:51.820494 (+ 945us) server_negotiation.cc:407] Connection header received
0619 11:09:51.821283 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:51.821286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:51.821332 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:51.821405 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:51.822520 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.823031 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.824110 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.824306 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.824374 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.826828 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:51.826831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:51.826833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:51.826856 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:51.828560 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.829076 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.829079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.829081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.829168 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:51.829457 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:51.829459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:51.829460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:51.829645 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:51.829718 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:51.830127 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 11:09:51.830211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:51.831070 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:51.819704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:44261 (local address 127.7.90.67:41509)
0619 11:09:51.820336 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:51.820374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:51.821114 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:51.821617 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:51.821624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:51.821947 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:51.822400 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.822413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.823128 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.823130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.824001 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:51.824010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:51.824415 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:51.824417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:51.824540 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:51.825077 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:51.825091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:51.826644 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:51.828666 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.828670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.828680 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.828980 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.829250 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:51.829252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:51.829253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:51.829374 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:51.829727 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:51.829731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:51.829943 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 11:09:51.830095 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:09:51.830306 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":122,"threads_started":1}
I20260619 11:09:51.831974 10428 heartbeater.cc:347] Connected to a master server at 127.7.90.126:44261
I20260619 11:09:51.832188 10428 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:51.832617 10428 heartbeater.cc:511] Master 127.7.90.126:44261 requested a full tablet report, sending...
I20260619 11:09:51.833571 9969 ts_manager.cc:194] Registered new tserver with Master: 21b33bdccb38469bb4557203f91d006c (127.7.90.67:44143)
I20260619 11:09:51.834136 9969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:41509
I20260619 11:09:51.840386 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:09:51.854247 10008 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:51.852007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.7.90.126:44261)
0619 11:09:51.852973 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:51.852977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:51.852999 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:09:51.853060 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:51.853064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:51.853112 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:51.853209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:51.854114 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41332: BlockingRecv error: recv got EOF from 127.0.0.1:41332 (error 108)
Metrics: {"server-negotiator.queue_time_us":879}
I20260619 11:09:51.855720 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10031
I20260619 11:09:51.861863 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10165
I20260619 11:09:51.867169 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10299
I20260619 11:09:51.873903 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 9937
2026-06-19T11:09:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2071 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[10440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[10440](info): set up 2 sockets
Jun 19 11:09:51 dist-test-slave-f1t5 krb5kdc[10440](info): commencing operation
krb5kdc: starting...
W20260619 11:09:52.897435 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:52 dist-test-slave-f1t5 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:53.012327 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:44597
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:53.117059 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:53.117295 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:53.117331 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:53.121209 10453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:53.121266 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:53.121285 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:53.121301 10453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:53.121316 10453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:53.126336 10453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:53.127210 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:53.128127 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:53.136298 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:53.136286 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:53.136286 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:53.136463 10453 server_base.cc:1034] running on GCE node
I20260619 11:09:53.137104 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:53.138180 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:53.139350 10453 hybrid_clock.cc:648] HybridClock initialized: now 1781867393139327 us; error 41 us; skew 500 ppm
Jun 19 11:09:53 dist-test-slave-f1t5 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867393, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:53.145875 10453 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:53.146782 10453 webserver.cc:460] Webserver started at http://127.7.90.126:39037/ using document root <none> and password file <none>
I20260619 11:09:53.147310 10453 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:53.147398 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:53.147621 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:53.149618 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "f088971a81ea483d943b93be76ad47de"
format_stamp: "Formatted at 2026-06-19 11:09:53 on dist-test-slave-f1t5"
server_key: "62884893b087c45751a36e22df552315"
server_key_iv: "a25eb7cc79377bd39983afc2daa6d920"
server_key_version: "encryptionkey@0"
I20260619 11:09:53.150151 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "f088971a81ea483d943b93be76ad47de"
format_stamp: "Formatted at 2026-06-19 11:09:53 on dist-test-slave-f1t5"
server_key: "62884893b087c45751a36e22df552315"
server_key_iv: "a25eb7cc79377bd39983afc2daa6d920"
server_key_version: "encryptionkey@0"
I20260619 11:09:53.153677 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:09:53.155869 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:53.156999 10453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:53.157140 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "f088971a81ea483d943b93be76ad47de"
format_stamp: "Formatted at 2026-06-19 11:09:53 on dist-test-slave-f1t5"
server_key: "62884893b087c45751a36e22df552315"
server_key_iv: "a25eb7cc79377bd39983afc2daa6d920"
server_key_version: "encryptionkey@0"
I20260619 11:09:53.157246 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:53.172398 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:53.175101 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:53.175292 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:53.181931 10453 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:44597
I20260619 11:09:53.182201 10520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:44597 every 8 connection(s)
I20260619 11:09:53.182875 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:53.185379 10521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:53.187739 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10453
I20260619 11:09:53.187855 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:53.188130 7529 external_mini_cluster.cc:1428] Setting key 48a262b99aadee7d7b894408f57f093f
I20260619 11:09:53.191455 10521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de: Bootstrap starting.
I20260619 11:09:53.193984 10521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:53.194746 10521 log.cc:824] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:53.196458 10521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de: No bootstrap required, opened a new log
Jun 19 11:09:53 dist-test-slave-f1t5 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:53.200321 10521 raft_consensus.cc:348] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } }
I20260619 11:09:53.200592 10521 raft_consensus.cc:374] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:53.200659 10521 raft_consensus.cc:729] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f088971a81ea483d943b93be76ad47de, State: Initialized, Role: FOLLOWER
I20260619 11:09:53.201133 10521 consensus_queue.cc:260] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } }
I20260619 11:09:53.201305 10521 raft_consensus.cc:388] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:53.201371 10521 raft_consensus.cc:482] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:53.201457 10521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:53.202454 10521 raft_consensus.cc:504] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } }
I20260619 11:09:53.202800 10521 leader_election.cc:302] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f088971a81ea483d943b93be76ad47de; no voters:
I20260619 11:09:53.203135 10521 leader_election.cc:288] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:53.203233 10526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:53.203440 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:53.189422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.7.90.126:44597)
0619 11:09:53.189875 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:53.189893 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:53.189934 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:09:53.190733 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:53.190756 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:53.191142 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:53.191509 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:53.192228 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:53.193696 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:53.194555 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:53.195207 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:53.195318 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:53.197866 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:53.197873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:53.197888 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:53.197917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:53.200385 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:53.200874 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:53.200881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:53.200885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:53.201014 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:53.201271 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:53.201274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:53.201275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:53.201790 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:53.201977 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:53.202187 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 11:09:53.202486 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260619 11:09:53.203467 10526 raft_consensus.cc:686] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [term 1 LEADER]: Becoming Leader. State: Replica: f088971a81ea483d943b93be76ad47de, State: Running, Role: LEADER
I20260619 11:09:53.203815 10526 consensus_queue.cc:237] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } }
I20260619 11:09:53.204273 10521 sys_catalog.cc:564] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:53.205498 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f088971a81ea483d943b93be76ad47de. Latest consensus state: current_term: 1 leader_uuid: "f088971a81ea483d943b93be76ad47de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } } }
I20260619 11:09:53.205632 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:53.206698 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:53.206496 10536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:53.206166 10527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f088971a81ea483d943b93be76ad47de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f088971a81ea483d943b93be76ad47de" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 44597 } } }
I20260619 11:09:53.207082 10527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:53.209647 10536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:53.214835 10536 catalog_manager.cc:1347] Generated new cluster ID: cf8e02e1b3e54ba49b0a94b1244609d7
I20260619 11:09:53.214905 10536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:53.234361 10536 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:53.235419 10536 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:53.236084 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:53.230829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.7.90.126:44597)
0619 11:09:53.230932 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:53.230937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:53.231165 (+ 228us) server_negotiation.cc:407] Connection header received
0619 11:09:53.231275 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:53.231278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:53.231326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:53.231409 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:53.231928 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:53.232709 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:53.233788 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:53.234105 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:53.234225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:53.235037 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:53.235041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:53.235044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:53.235597 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:53.235630 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:53.235656 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:53.235744 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:53.235878 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 11:09:53.235950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:09:53.250895 10536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f088971a81ea483d943b93be76ad47de: Generated new TSK 0
I20260619 11:09:53.251525 10536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:53.257658 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10453
2026-06-19T11:09:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1381 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:53 dist-test-slave-f1t5 krb5kdc[10554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:53 dist-test-slave-f1t5 krb5kdc[10554](info): set up 2 sockets
Jun 19 11:09:53 dist-test-slave-f1t5 krb5kdc[10554](info): commencing operation
krb5kdc: starting...
W20260619 11:09:54.285480 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:54.399783 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38155
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:54.505023 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:54.505298 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:54.505376 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:54.509200 10567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:54.509275 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:54.509308 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:54.509341 10567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:54.509370 10567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:54.515588 10567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:54.516573 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:54.517617 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:54.525695 10573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:54.525852 10575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.525918 10567 server_base.cc:1034] running on GCE node
W20260619 11:09:54.525695 10572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.526485 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:54.527556 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:54.528723 10567 hybrid_clock.cc:648] HybridClock initialized: now 1781867394528713 us; error 26 us; skew 500 ppm
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:54.531373 10567 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:54.532193 10567 webserver.cc:460] Webserver started at http://127.7.90.126:40039/ using document root <none> and password file <none>
I20260619 11:09:54.532724 10567 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:54.532812 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:54.533033 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:54.535118 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "09b688ea0d214650af33ba273785637c"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f1ddd6b9b8122dfb7f364d303a6bceda"
server_key_iv: "1236ef616b1a79531aa065e21f133c3e"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.535673 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "09b688ea0d214650af33ba273785637c"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f1ddd6b9b8122dfb7f364d303a6bceda"
server_key_iv: "1236ef616b1a79531aa065e21f133c3e"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.539157 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:09:54.541428 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:54.542682 10567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:54.542822 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "09b688ea0d214650af33ba273785637c"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f1ddd6b9b8122dfb7f364d303a6bceda"
server_key_iv: "1236ef616b1a79531aa065e21f133c3e"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.542934 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:54.555071 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:54.558216 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:54.558411 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:54.565259 10567 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:38155
I20260619 11:09:54.565553 10634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:38155 every 8 connection(s)
I20260619 11:09:54.566237 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:54.568816 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:54.574200 10635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c: Bootstrap starting.
I20260619 11:09:54.575217 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10567
I20260619 11:09:54.575306 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:54.575560 7529 external_mini_cluster.cc:1428] Setting key dbf7fc93923807d1551c671a1041e4f0
I20260619 11:09:54.576890 10635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:54.577777 10635 log.cc:824] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:54.579825 10635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c: No bootstrap required, opened a new log
I20260619 11:09:54.583745 10635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } }
I20260619 11:09:54.583943 10635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:54.583985 10635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b688ea0d214650af33ba273785637c, State: Initialized, Role: FOLLOWER
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:54.584393 10635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } }
I20260619 11:09:54.584497 10635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:54.584578 10635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:54.584692 10635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:54.585829 10635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } }
I20260619 11:09:54.586195 10635 leader_election.cc:302] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b688ea0d214650af33ba273785637c; no voters:
I20260619 11:09:54.586553 10635 leader_election.cc:288] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:54.586653 10640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:54.586894 10640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [term 1 LEADER]: Becoming Leader. State: Replica: 09b688ea0d214650af33ba273785637c, State: Running, Role: LEADER
I20260619 11:09:54.587242 10640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } }
I20260619 11:09:54.587800 10635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:54.588938 10642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b688ea0d214650af33ba273785637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } } }
I20260619 11:09:54.589056 10642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:54.589367 10641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b688ea0d214650af33ba273785637c. Latest consensus state: current_term: 1 leader_uuid: "09b688ea0d214650af33ba273785637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b688ea0d214650af33ba273785637c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38155 } } }
I20260619 11:09:54.589457 10649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:54.589445 10641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:54.590931 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:54.576629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.7.90.126:38155)
0619 11:09:54.577080 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:54.577100 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:54.577153 (+ 53us) server_negotiation.cc:407] Connection header received
0619 11:09:54.578031 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:54.578056 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:54.578401 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:54.578779 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:54.579400 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.580829 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.581701 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.582412 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.582620 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:54.585055 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:54.585060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:54.585075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:54.585103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:54.587129 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:54.587613 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:54.587619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:54.587624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:54.587757 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:54.588053 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:54.588056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:54.588058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:54.588575 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:54.588787 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:54.589660 (+ 873us) server_negotiation.cc:299] Negotiation successful
0619 11:09:54.589971 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":129,"threads_started":1}
I20260619 11:09:54.592473 10649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:54.597967 10649 catalog_manager.cc:1347] Generated new cluster ID: 705b6893e62d4096a749e7c1b0738303
I20260619 11:09:54.598037 10649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:54.613919 10649 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:54.614796 10649 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:54.621655 10649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b688ea0d214650af33ba273785637c: Generated new TSK 0
I20260619 11:09:54.622268 10649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:54.649698 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:54.757099 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:54.757323 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:54.757357 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:54.761266 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:54.761320 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:54.761375 10661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:54.766536 10661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:54.767408 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:54.768299 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:54.777865 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.777944 10661 server_base.cc:1034] running on GCE node
W20260619 11:09:54.777832 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:54.777886 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.778389 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:54.778992 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:54.780169 10661 hybrid_clock.cc:648] HybridClock initialized: now 1781867394780164 us; error 30 us; skew 500 ppm
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:54.782891 10661 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:54.783638 10661 webserver.cc:460] Webserver started at http://127.7.90.65:37479/ using document root <none> and password file <none>
I20260619 11:09:54.784157 10661 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:54.784245 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:54.784454 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:54.786358 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "6a63510528a349b89b9376fe882ad528"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "7fae18ff6d3746d3f029553478bf0d36"
server_key_iv: "2722075cc9647e61bd5e4d659c3ccd48"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.786931 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "6a63510528a349b89b9376fe882ad528"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "7fae18ff6d3746d3f029553478bf0d36"
server_key_iv: "2722075cc9647e61bd5e4d659c3ccd48"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.790591 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:09:54.792747 10676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:54.793951 10661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:09:54.794073 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "6a63510528a349b89b9376fe882ad528"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "7fae18ff6d3746d3f029553478bf0d36"
server_key_iv: "2722075cc9647e61bd5e4d659c3ccd48"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.794175 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:54.810410 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:54.813544 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:54.813758 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:54.814284 10661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:54.815032 10661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:54.815105 10661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:54.815176 10661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:54.815217 10661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:54.823626 10661 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:40147
I20260619 11:09:54.823652 10789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:40147 every 8 connection(s)
I20260619 11:09:54.824451 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:54.825131 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10661
I20260619 11:09:54.825245 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:54.825536 7529 external_mini_cluster.cc:1428] Setting key 558432d5471d6cf9da037f1e5295271c
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:09:54.839097 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:54.826237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:52017 (local address 127.7.90.126:38155)
0619 11:09:54.826348 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:54.826352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:54.827331 (+ 979us) server_negotiation.cc:407] Connection header received
0619 11:09:54.829263 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:54.829269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:54.829331 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:54.829436 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:54.830518 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.831402 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.832457 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.832850 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.832994 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:54.835658 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:54.835661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:54.835663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:54.835686 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:54.837045 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:54.837571 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:54.837575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:54.837576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:54.837656 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:54.838000 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:54.838006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:54.838011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:54.838259 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:54.838371 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:54.838798 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 11:09:54.838937 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:09:54.840030 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:54.826678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38155 (local address 127.7.90.65:52017)
0619 11:09:54.827176 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:54.827228 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:54.829072 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:54.829620 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:54.829626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:54.829926 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:54.830374 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:54.830386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.831517 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.831519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:54.832338 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:54.832346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:54.833016 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:54.833018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:54.833153 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:54.833807 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:54.833822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:54.835504 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:54.837139 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:54.837144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:54.837153 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:54.837426 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:54.837748 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:54.837754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:54.837756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:54.837891 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:54.838364 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:54.838368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:54.838567 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 11:09:54.838719 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:09:54.839038 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":145,"threads_started":1}
I20260619 11:09:54.841202 10790 heartbeater.cc:347] Connected to a master server at 127.7.90.126:38155
I20260619 11:09:54.841434 10790 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:54.841975 10790 heartbeater.cc:511] Master 127.7.90.126:38155 requested a full tablet report, sending...
I20260619 11:09:54.843415 10599 ts_manager.cc:194] Registered new tserver with Master: 6a63510528a349b89b9376fe882ad528 (127.7.90.65:40147)
I20260619 11:09:54.844947 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:52017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:54.852833 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:54.956092 10795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:54.956359 10795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:54.956432 10795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:54.960300 10795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:54.960378 10795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:54.960459 10795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:54.965229 10795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:54.966192 10795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:54.967119 10795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:54.976392 10800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:54.976455 10801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.976564 10795 server_base.cc:1034] running on GCE node
W20260619 11:09:54.976455 10803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:54.977140 10795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:54.977783 10795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:54.978945 10795 hybrid_clock.cc:648] HybridClock initialized: now 1781867394978928 us; error 30 us; skew 500 ppm
Jun 19 11:09:54 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:54.981951 10795 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:54.983036 10795 webserver.cc:460] Webserver started at http://127.7.90.66:36641/ using document root <none> and password file <none>
I20260619 11:09:54.983690 10795 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:54.983793 10795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:54.984073 10795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:54.986526 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "4db116454ba94d648616857f0fa7e394"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f3da8f2b2c2de07496e8f15651ee4b0b"
server_key_iv: "f1fc2c270431220445785ea39f9a8233"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.987210 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "4db116454ba94d648616857f0fa7e394"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f3da8f2b2c2de07496e8f15651ee4b0b"
server_key_iv: "f1fc2c270431220445785ea39f9a8233"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.991497 10795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:09:54.993913 10810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:54.995097 10795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:54.995234 10795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "4db116454ba94d648616857f0fa7e394"
format_stamp: "Formatted at 2026-06-19 11:09:54 on dist-test-slave-f1t5"
server_key: "f3da8f2b2c2de07496e8f15651ee4b0b"
server_key_iv: "f1fc2c270431220445785ea39f9a8233"
server_key_version: "encryptionkey@0"
I20260619 11:09:54.995343 10795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:55.023496 10795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:55.026571 10795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:55.026785 10795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:55.027383 10795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:55.028282 10795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:55.028350 10795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:55.028425 10795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:55.028470 10795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:55.036980 10795 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45755
I20260619 11:09:55.037001 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45755 every 8 connection(s)
I20260619 11:09:55.037860 10795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:55.038617 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10795
I20260619 11:09:55.038717 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:55.039009 7529 external_mini_cluster.cc:1428] Setting key d9f0a5010607ca5ebcc2db7c7bc46121
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:09:55.051954 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:55.039689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:40293 (local address 127.7.90.126:38155)
0619 11:09:55.039829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:55.039833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:55.040824 (+ 991us) server_negotiation.cc:407] Connection header received
0619 11:09:55.042657 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:55.042660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:55.042714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:55.042796 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:55.043914 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.044433 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.045564 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.045790 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.045901 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:55.048357 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:55.048361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:55.048364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:55.048394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:55.049902 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.050407 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.050412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.050413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.050502 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.050907 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.050910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.050912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.051089 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:55.051196 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:55.051667 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 11:09:55.051783 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:09:55.052816 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:55.040124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38155 (local address 127.7.90.66:40293)
0619 11:09:55.040665 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:55.040717 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:55.042451 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:55.042960 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:55.042968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:55.043313 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:55.043774 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:55.043786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.044555 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.044560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:55.045420 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:55.045428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.045935 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.045940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:55.046116 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:55.046699 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:55.046715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:55.048204 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:55.050029 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:55.050035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:55.050049 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:55.050303 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:55.050613 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:55.050617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:55.050619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:55.050808 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:55.051191 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:55.051197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:55.051437 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 11:09:55.051628 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 11:09:55.051936 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20260619 11:09:55.054136 10924 heartbeater.cc:347] Connected to a master server at 127.7.90.126:38155
I20260619 11:09:55.054382 10924 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:55.054850 10924 heartbeater.cc:511] Master 127.7.90.126:38155 requested a full tablet report, sending...
I20260619 11:09:55.055933 10599 ts_manager.cc:194] Registered new tserver with Master: 4db116454ba94d648616857f0fa7e394 (127.7.90.66:45755)
I20260619 11:09:55.056788 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:40293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:55.066504 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:55.171881 10929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:55.172092 10929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:55.172129 10929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:55.176384 10929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:55.176433 10929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:55.176491 10929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:55.181293 10929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:55.182127 10929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:55.182977 10929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:55.192183 10935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:55.192188 10934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:55.192241 10937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:55.192382 10929 server_base.cc:1034] running on GCE node
I20260619 11:09:55.192775 10929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:55.193315 10929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:55.194489 10929 hybrid_clock.cc:648] HybridClock initialized: now 1781867395194458 us; error 48 us; skew 500 ppm
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867395, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:55.197099 10929 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:55.197954 10929 webserver.cc:460] Webserver started at http://127.7.90.67:38649/ using document root <none> and password file <none>
I20260619 11:09:55.198484 10929 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:55.198542 10929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:55.198714 10929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:55.200593 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "c5285f5beb3542bc9fd88630b1860136"
format_stamp: "Formatted at 2026-06-19 11:09:55 on dist-test-slave-f1t5"
server_key: "b165d00b2bef63d76292b19057212647"
server_key_iv: "ee37ffb7f34ebb55874eee7827e4b158"
server_key_version: "encryptionkey@0"
I20260619 11:09:55.201052 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "c5285f5beb3542bc9fd88630b1860136"
format_stamp: "Formatted at 2026-06-19 11:09:55 on dist-test-slave-f1t5"
server_key: "b165d00b2bef63d76292b19057212647"
server_key_iv: "ee37ffb7f34ebb55874eee7827e4b158"
server_key_version: "encryptionkey@0"
I20260619 11:09:55.204367 10929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:55.206650 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:55.207767 10929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:55.207887 10929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "c5285f5beb3542bc9fd88630b1860136"
format_stamp: "Formatted at 2026-06-19 11:09:55 on dist-test-slave-f1t5"
server_key: "b165d00b2bef63d76292b19057212647"
server_key_iv: "ee37ffb7f34ebb55874eee7827e4b158"
server_key_version: "encryptionkey@0"
I20260619 11:09:55.207979 10929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:55.234483 10929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:55.237589 10929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:55.237780 10929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:55.238314 10929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:55.239059 10929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:55.239110 10929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:55.239174 10929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:55.239212 10929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:55.247782 10929 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:38059
I20260619 11:09:55.247794 11057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:38059 every 8 connection(s)
I20260619 11:09:55.248595 10929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:55.252313 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 10929
I20260619 11:09:55.252455 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:55.252753 7529 external_mini_cluster.cc:1428] Setting key 9b4ffa2101c549fd48b89bba7d0b0c6d
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867395, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:55.261559 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:55.250278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:53723 (local address 127.7.90.126:38155)
0619 11:09:55.250400 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:55.250403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:55.251210 (+ 807us) server_negotiation.cc:407] Connection header received
0619 11:09:55.252674 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:55.252677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:55.252721 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:55.252783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:55.254306 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.254918 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.256029 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.256244 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.256346 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:55.258766 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:55.258769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:55.258772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:55.258793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:55.259946 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.260359 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.260361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.260362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.260429 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.260696 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.260698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.260699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.260855 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:55.260921 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:55.261313 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 11:09:55.261399 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:55.262279 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:55.250654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38155 (local address 127.7.90.67:53723)
0619 11:09:55.251068 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:55.251107 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:55.252433 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:55.252979 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:55.252987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:55.253447 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:55.254174 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:55.254189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.255030 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.255034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:55.255909 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:55.255917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.256379 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.256383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:55.256573 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:55.257167 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:55.257182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:55.258624 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:55.260036 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:55.260040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:55.260050 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:55.260281 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:55.260507 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:55.260509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:55.260510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:55.260626 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:55.260930 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:55.260934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:55.261123 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 11:09:55.261255 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 11:09:55.261486 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260619 11:09:55.263336 11058 heartbeater.cc:347] Connected to a master server at 127.7.90.126:38155
I20260619 11:09:55.263528 11058 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:55.263917 11058 heartbeater.cc:511] Master 127.7.90.126:38155 requested a full tablet report, sending...
I20260619 11:09:55.264823 10599 ts_manager.cc:194] Registered new tserver with Master: c5285f5beb3542bc9fd88630b1860136 (127.7.90.67:38059)
I20260619 11:09:55.265480 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:53723
I20260619 11:09:55.266392 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:55.291018 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:55.282732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.7.90.126:38155)
0619 11:09:55.282882 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:55.282888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:55.283052 (+ 164us) server_negotiation.cc:407] Connection header received
0619 11:09:55.283207 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:55.283210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:55.283260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:55.283346 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:55.284018 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.284530 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.285355 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:55.285645 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:55.285754 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:55.288059 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:55.288062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:55.288065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:55.288096 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:55.289351 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.289803 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.289807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.289809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.289893 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:55.290164 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:55.290169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:55.290171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:55.290378 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:55.290524 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:55.290740 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 11:09:55.290853 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 11:09:55.291381 10599 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48240
I20260619 11:09:55.292112 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10661
I20260619 11:09:55.297920 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10795
I20260619 11:09:55.303493 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10929
I20260619 11:09:55.308586 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 10567
2026-06-19T11:09:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2053 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[11070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[11070](info): set up 2 sockets
Jun 19 11:09:55 dist-test-slave-f1t5 krb5kdc[11070](info): commencing operation
krb5kdc: starting...
W20260619 11:09:56.337457 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:56 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:56.449004 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41657
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:56.555387 11083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:56.555634 11083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:56.555688 11083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:56.559497 11083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:56.559543 11083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:56.559563 11083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:56.559579 11083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:56.559594 11083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:56.565156 11083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:41657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:56.566200 11083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:56.567126 11083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:56.575276 11089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:56.575276 11091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:56.575448 11088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:56.575680 11083 server_base.cc:1034] running on GCE node
I20260619 11:09:56.576290 11083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:56.577348 11083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:56.578517 11083 hybrid_clock.cc:648] HybridClock initialized: now 1781867396578515 us; error 63 us; skew 500 ppm
Jun 19 11:09:56 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867396, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:56.581195 11083 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:56.582023 11083 webserver.cc:460] Webserver started at http://127.7.90.126:34409/ using document root <none> and password file <none>
I20260619 11:09:56.582509 11083 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:56.582563 11083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:56.582738 11083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:56.584616 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "f89d550805e4fed2ef0ee86640197df7"
server_key_iv: "57e5d8218ab6ef3e22aa8ba94130c666"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.585085 11083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "f89d550805e4fed2ef0ee86640197df7"
server_key_iv: "57e5d8218ab6ef3e22aa8ba94130c666"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.588421 11083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:56.590603 11098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:56.591706 11083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:56.591792 11083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "f89d550805e4fed2ef0ee86640197df7"
server_key_iv: "57e5d8218ab6ef3e22aa8ba94130c666"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.591862 11083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:56.602447 11083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:56.605434 11083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:56.605643 11083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:56.611897 11083 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:41657
I20260619 11:09:56.611913 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:41657 every 8 connection(s)
I20260619 11:09:56.612726 11083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:56.614280 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11083
I20260619 11:09:56.614395 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:56.614678 7529 external_mini_cluster.cc:1428] Setting key d2b77f222fced4f8c524c24c6a3357dd
I20260619 11:09:56.615682 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:56.620832 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2: Bootstrap starting.
Jun 19 11:09:56 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:56.622845 11151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:56.623513 11151 log.cc:824] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:56.625566 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2: No bootstrap required, opened a new log
I20260619 11:09:56.627876 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:56.615943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.7.90.126:41657)
0619 11:09:56.616329 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:56.616342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:56.616376 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:09:56.616920 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:56.616933 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:56.617256 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:56.617568 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:56.618348 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.619298 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.620145 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.620578 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.620740 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:56.622983 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:56.622988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:56.622998 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:56.623022 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:56.624913 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:56.625314 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:56.625321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:56.625327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:56.625460 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:56.625797 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:56.625801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:56.625802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:56.626257 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:56.626435 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:56.626657 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 11:09:56.626937 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260619 11:09:56.629791 11151 raft_consensus.cc:348] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } }
I20260619 11:09:56.630057 11151 raft_consensus.cc:374] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:56.630154 11151 raft_consensus.cc:729] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89a28e0a9ad42c6b6a55b4f4bc687d2, State: Initialized, Role: FOLLOWER
I20260619 11:09:56.630595 11151 consensus_queue.cc:260] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } }
I20260619 11:09:56.630738 11151 raft_consensus.cc:388] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:56.630823 11151 raft_consensus.cc:482] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:56.630916 11151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:56.631700 11151 raft_consensus.cc:504] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } }
I20260619 11:09:56.632016 11151 leader_election.cc:302] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d89a28e0a9ad42c6b6a55b4f4bc687d2; no voters:
I20260619 11:09:56.632309 11151 leader_election.cc:288] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:56.632452 11156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:56.632699 11156 raft_consensus.cc:686] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [term 1 LEADER]: Becoming Leader. State: Replica: d89a28e0a9ad42c6b6a55b4f4bc687d2, State: Running, Role: LEADER
I20260619 11:09:56.633191 11156 consensus_queue.cc:237] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } }
I20260619 11:09:56.633504 11151 sys_catalog.cc:564] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:56.634281 11157 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d89a28e0a9ad42c6b6a55b4f4bc687d2. Latest consensus state: current_term: 1 leader_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } } }
I20260619 11:09:56.634389 11157 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:56.634730 11165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:56.635116 11158 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89a28e0a9ad42c6b6a55b4f4bc687d2" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41657 } } }
I20260619 11:09:56.635277 11158 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:56.637244 11165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:56.642117 11165 catalog_manager.cc:1347] Generated new cluster ID: 6ece51a3fbec42a1a29c9ea0803d0bdb
I20260619 11:09:56.642180 11165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:56.659540 11165 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:56.660358 11165 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:56.666889 11165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d89a28e0a9ad42c6b6a55b4f4bc687d2: Generated new TSK 0
I20260619 11:09:56.667526 11165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:56.704466 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:56.829537 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:56.829777 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:56.829849 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:56.833891 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:56.833961 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:56.834049 11177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:56.839237 11177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:56.840232 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:56.841212 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:56.850338 11182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:56.850338 11183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:56.850373 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:56.850716 11177 server_base.cc:1034] running on GCE node
I20260619 11:09:56.851071 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:56.851684 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:56.852839 11177 hybrid_clock.cc:648] HybridClock initialized: now 1781867396852818 us; error 34 us; skew 500 ppm
Jun 19 11:09:56 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867396, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:56.855489 11177 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:56.856276 11177 webserver.cc:460] Webserver started at http://127.7.90.65:34839/ using document root <none> and password file <none>
I20260619 11:09:56.856760 11177 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:56.856855 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:56.857070 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:56.859035 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "be99e43a6dba45ca8461e2b9401cd25e"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "6da0fa77467b96188afb71b832dbda7e"
server_key_iv: "7fe0e94f79ea0aa70b694887976cb31d"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.859594 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "be99e43a6dba45ca8461e2b9401cd25e"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "6da0fa77467b96188afb71b832dbda7e"
server_key_iv: "7fe0e94f79ea0aa70b694887976cb31d"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.863699 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:09:56.866050 11192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:56.867197 11177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:56.867323 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "be99e43a6dba45ca8461e2b9401cd25e"
format_stamp: "Formatted at 2026-06-19 11:09:56 on dist-test-slave-f1t5"
server_key: "6da0fa77467b96188afb71b832dbda7e"
server_key_iv: "7fe0e94f79ea0aa70b694887976cb31d"
server_key_version: "encryptionkey@0"
I20260619 11:09:56.867426 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:56.880505 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:56.883611 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:56.883798 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:56.884321 11177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:56.885061 11177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:56.885141 11177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:56.885207 11177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:56.885262 11177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:56.893703 11177 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36107
I20260619 11:09:56.893725 11305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36107 every 8 connection(s)
I20260619 11:09:56.894567 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:56.900534 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11177
I20260619 11:09:56.900687 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:56.900990 7529 external_mini_cluster.cc:1428] Setting key 478ad05d6c51bc32a0d15b9218f1f054
Jun 19 11:09:56 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867396, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:56.908051 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:56.896277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:47931 (local address 127.7.90.126:41657)
0619 11:09:56.896406 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:56.896410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:56.897188 (+ 778us) server_negotiation.cc:407] Connection header received
0619 11:09:56.898580 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:56.898583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:56.898641 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:56.898744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:56.899820 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.900444 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.901823 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.902052 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.902177 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:56.904573 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:56.904578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:56.904581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:56.904610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:56.906040 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:56.906564 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:56.906569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:56.906571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:56.906651 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:56.906977 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:56.906981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:56.906983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:56.907165 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:56.907279 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:56.907767 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 11:09:56.907885 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:09:56.908795 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:56.896628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41657 (local address 127.7.90.65:47931)
0619 11:09:56.897045 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:56.897083 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:56.898379 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:56.898906 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:56.898913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:56.899239 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:56.899675 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:56.899687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.900628 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.900631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:56.901695 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:56.901707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:56.902247 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:56.902251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:56.902450 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:56.903040 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:56.903055 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:56.904413 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:56.906178 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:56.906183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:56.906197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:56.906461 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:56.906743 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:56.906747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:56.906749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:56.906883 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:56.907287 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:56.907296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:56.907538 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 11:09:56.907727 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 11:09:56.908021 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:09:56.910132 11306 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41657
I20260619 11:09:56.910354 11306 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:56.910795 11306 heartbeater.cc:511] Master 127.7.90.126:41657 requested a full tablet report, sending...
I20260619 11:09:56.912238 11114 ts_manager.cc:194] Registered new tserver with Master: be99e43a6dba45ca8461e2b9401cd25e (127.7.90.65:36107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:09:56.913335 11114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:47931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:56.926645 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:57.031733 11311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:57.031940 11311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:57.031978 11311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:57.036046 11311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:57.036093 11311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:57.036185 11311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:57.041162 11311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:57.042129 11311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:57.043004 11311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:57.052358 11316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:57.052363 11319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:57.052465 11311 server_base.cc:1034] running on GCE node
W20260619 11:09:57.052358 11317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:57.053018 11311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:57.053671 11311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:57.054853 11311 hybrid_clock.cc:648] HybridClock initialized: now 1781867397054839 us; error 24 us; skew 500 ppm
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:57.057441 11311 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:57.058264 11311 webserver.cc:460] Webserver started at http://127.7.90.66:45127/ using document root <none> and password file <none>
I20260619 11:09:57.058794 11311 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:57.058887 11311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:57.059113 11311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:57.061025 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "46ec4a7c3533488c95862a3c0723088e"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "4c518069c2867defedd9347512498d75"
server_key_iv: "66b52826c8a9defaad2ddba1f89d5582"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.061640 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "46ec4a7c3533488c95862a3c0723088e"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "4c518069c2867defedd9347512498d75"
server_key_iv: "66b52826c8a9defaad2ddba1f89d5582"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.065063 11311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:57.067374 11326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.068516 11311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:09:57.068662 11311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "46ec4a7c3533488c95862a3c0723088e"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "4c518069c2867defedd9347512498d75"
server_key_iv: "66b52826c8a9defaad2ddba1f89d5582"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.068768 11311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:57.106617 11311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:57.109903 11311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:57.110106 11311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:57.110653 11311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:57.111476 11311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:57.111547 11311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.111624 11311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:57.111670 11311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.120648 11311 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45701
I20260619 11:09:57.120666 11439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45701 every 8 connection(s)
I20260619 11:09:57.121480 11311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:57.122129 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11311
I20260619 11:09:57.122243 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:57.122476 7529 external_mini_cluster.cc:1428] Setting key 667baa43e8ac57c5c7f31e5f3863a75f
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:09:57.136224 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.124115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:35819 (local address 127.7.90.126:41657)
0619 11:09:57.124259 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:57.124265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:57.125404 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 11:09:57.126282 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:57.126285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:57.126333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:57.126405 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:57.127505 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.128388 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.129589 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.130032 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.130156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.132401 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:57.132408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:57.132415 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:57.132444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:57.134068 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.134499 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.134501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.134502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.134857 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.135162 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.135164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.135165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.135370 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:57.135475 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:57.136013 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 11:09:57.136107 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:57.137382 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.124617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41657 (local address 127.7.90.66:35819)
0619 11:09:57.125272 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:57.125309 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:57.126094 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:57.126558 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:57.126565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:57.126910 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:57.127358 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:57.127370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.128522 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.128525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:57.129442 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:57.129451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.130165 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.130167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:57.130300 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.130860 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:57.130885 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:57.132223 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:57.134174 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:57.134178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:57.134188 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:57.134417 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:57.134954 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:57.134956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:57.134958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:57.135081 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:57.135481 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:57.135489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:57.135757 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 11:09:57.135993 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 11:09:57.136347 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":132,"threads_started":1}
I20260619 11:09:57.138571 11440 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41657
I20260619 11:09:57.138798 11440 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:57.139274 11440 heartbeater.cc:511] Master 127.7.90.126:41657 requested a full tablet report, sending...
I20260619 11:09:57.140309 11114 ts_manager.cc:194] Registered new tserver with Master: 46ec4a7c3533488c95862a3c0723088e (127.7.90.66:45701)
I20260619 11:09:57.140861 11114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:35819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:57.150434 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:57.256290 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:57.256497 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:57.256536 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:57.260514 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:57.260563 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:57.260686 11445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:57.265916 11445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:57.266871 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:57.267808 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:57.276854 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:57.276887 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:57.276883 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:57.276970 11445 server_base.cc:1034] running on GCE node
I20260619 11:09:57.277612 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:57.278234 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:57.279403 11445 hybrid_clock.cc:648] HybridClock initialized: now 1781867397279395 us; error 36 us; skew 500 ppm
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:57.282052 11445 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:57.282819 11445 webserver.cc:460] Webserver started at http://127.7.90.67:44631/ using document root <none> and password file <none>
I20260619 11:09:57.283350 11445 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:57.283437 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:57.283630 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:57.285457 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "f2fe4d4861ab48b2bbcf897ef9ae332b"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "88b802024f8157c25cac385028493567"
server_key_iv: "139699678478ebdb621d7375b82056d1"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.286041 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "f2fe4d4861ab48b2bbcf897ef9ae332b"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "88b802024f8157c25cac385028493567"
server_key_iv: "139699678478ebdb621d7375b82056d1"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.289372 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:57.291554 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.292636 11445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:57.292759 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "f2fe4d4861ab48b2bbcf897ef9ae332b"
format_stamp: "Formatted at 2026-06-19 11:09:57 on dist-test-slave-f1t5"
server_key: "88b802024f8157c25cac385028493567"
server_key_iv: "139699678478ebdb621d7375b82056d1"
server_key_version: "encryptionkey@0"
I20260619 11:09:57.292876 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:57.325884 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:57.328549 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:57.328739 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:57.329298 11445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:57.330111 11445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:57.330178 11445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.330246 11445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:57.330298 11445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:57.339092 11445 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:40445
I20260619 11:09:57.339109 11573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:40445 every 8 connection(s)
I20260619 11:09:57.339898 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:57.346324 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11445
I20260619 11:09:57.346429 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:57.346726 7529 external_mini_cluster.cc:1428] Setting key a292282865ab7de87686127a02631f4d
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:57.352203 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.341510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:45289 (local address 127.7.90.126:41657)
0619 11:09:57.341663 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:57.341666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:57.342407 (+ 741us) server_negotiation.cc:407] Connection header received
0619 11:09:57.343784 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:57.343787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:57.343844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:57.343915 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:57.344985 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.345511 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.346765 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.347038 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.347150 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.349333 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:57.349336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:57.349338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:57.349361 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:57.350553 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.351034 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.351037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.351038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.351105 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.351376 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.351378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.351379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.351534 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:57.351604 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:57.352011 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 11:09:57.352095 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:09:57.352923 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.341874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41657 (local address 127.7.90.67:45289)
0619 11:09:57.342278 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:57.342316 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:57.343574 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:57.344072 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:57.344085 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:57.344422 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:57.344852 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:57.344864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.345691 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.345693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:57.346622 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:57.346630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.347180 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.347184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:57.347353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.347825 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:57.347840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:57.349197 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:57.350659 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:57.350663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:57.350691 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:57.350931 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:57.351181 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:57.351183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:57.351185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:57.351303 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:57.351622 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:57.351626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:57.351821 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 11:09:57.351961 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 11:09:57.352175 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260619 11:09:57.353946 11574 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41657
I20260619 11:09:57.354163 11574 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:57.354590 11574 heartbeater.cc:511] Master 127.7.90.126:41657 requested a full tablet report, sending...
I20260619 11:09:57.355486 11114 ts_manager.cc:194] Registered new tserver with Master: f2fe4d4861ab48b2bbcf897ef9ae332b (127.7.90.67:40445)
I20260619 11:09:57.356043 11114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:45289
I20260619 11:09:57.360348 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:09:57.384837 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.375883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.7.90.65:36107)
0619 11:09:57.376142 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:57.376156 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:57.376181 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:09:57.376229 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:57.376233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:57.376409 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:57.376523 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:57.377130 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.377958 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.378728 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.379065 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.379142 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.381486 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:57.381494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:57.381505 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:57.381566 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:57.383139 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.383563 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.383569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.383570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.383654 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.383922 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.383926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.383928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.384151 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:57.384369 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:57.384527 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 11:09:57.384678 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:57.394361 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:57.387359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.7.90.126:41657)
0619 11:09:57.387498 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:57.387502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:57.387515 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:57.387579 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:57.387581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:57.387615 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:57.387673 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:57.388176 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.388651 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.389339 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:57.389571 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:57.389649 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:57.391782 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:57.391785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:57.391786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:57.391807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:57.392927 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.393261 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.393263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.393264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.393329 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:57.393569 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:57.393572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:57.393573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:57.393789 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:57.393891 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:57.394067 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 11:09:57.394151 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 11:09:57.394840 11114 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59472
I20260619 11:09:57.395428 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11177
I20260619 11:09:57.400605 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11311
I20260619 11:09:57.405850 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11445
I20260619 11:09:57.410772 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11083
2026-06-19T11:09:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2102 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11586](info): set up 2 sockets
Jun 19 11:09:57 dist-test-slave-f1t5 krb5kdc[11586](info): commencing operation
krb5kdc: starting...
W20260619 11:09:58.437611 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:09:58 dist-test-slave-f1t5 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:09:58.557045 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37323
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:58.661311 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:58.661625 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:58.661736 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:58.665705 11599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:58.665788 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:58.665825 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:58.665860 11599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:58.665891 11599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:58.670965 11599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:58.672034 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:58.673018 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:58.681284 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:58.681345 11599 server_base.cc:1034] running on GCE node
W20260619 11:09:58.681278 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:58.681277 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:58.682080 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:58.683254 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:58.684413 11599 hybrid_clock.cc:648] HybridClock initialized: now 1781867398684376 us; error 50 us; skew 500 ppm
Jun 19 11:09:58 dist-test-slave-f1t5 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867398, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:58.691107 11599 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:09:58.691869 11599 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:58.691933 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:58.692101 11599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:58.694622 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "f752330fa73a4ecab320dd0ad5a32238"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "c3a8716aa669e4801fdd683c624eac34"
server_key_iv: "2fa5b5d234b78599092f1d78a7319015"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.695130 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "f752330fa73a4ecab320dd0ad5a32238"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "c3a8716aa669e4801fdd683c624eac34"
server_key_iv: "2fa5b5d234b78599092f1d78a7319015"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.698643 11599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:09:58.700933 11613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:58.702159 11599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:58.702282 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "f752330fa73a4ecab320dd0ad5a32238"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "c3a8716aa669e4801fdd683c624eac34"
server_key_iv: "2fa5b5d234b78599092f1d78a7319015"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.702378 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:58.721825 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:58.724423 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:58.724620 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:58.731387 11599 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37323
I20260619 11:09:58.731652 11665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37323 every 8 connection(s)
I20260619 11:09:58.732021 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:58.733003 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11599
I20260619 11:09:58.733121 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:58.733379 7529 external_mini_cluster.cc:1428] Setting key e9825b408c43ceaa35f742164864861e
I20260619 11:09:58.735049 11666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:58.741686 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: Bootstrap starting.
Jun 19 11:09:58 dist-test-slave-f1t5 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:58.744181 11666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:58.745067 11666 log.cc:824] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:58.746933 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: No bootstrap required, opened a new log
I20260619 11:09:58.747761 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:58.734774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.7.90.126:37323)
0619 11:09:58.735183 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:58.735201 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:58.735242 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:09:58.736054 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:58.736076 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:58.736476 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:58.736832 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:58.737475 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.738975 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.739800 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.740505 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.740659 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:58.743036 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:58.743042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:58.743056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:58.743084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:58.744974 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:58.745413 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:58.745422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:58.745428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:58.745609 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:58.745895 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:58.745898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:58.745900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:58.746267 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:58.746470 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:58.746746 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 11:09:58.747062 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260619 11:09:58.750466 11666 raft_consensus.cc:348] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } }
I20260619 11:09:58.750672 11666 raft_consensus.cc:374] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:58.750742 11666 raft_consensus.cc:729] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f752330fa73a4ecab320dd0ad5a32238, State: Initialized, Role: FOLLOWER
I20260619 11:09:58.751183 11666 consensus_queue.cc:260] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } }
I20260619 11:09:58.751322 11666 raft_consensus.cc:388] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:58.751403 11666 raft_consensus.cc:482] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:58.751502 11666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:58.752283 11666 raft_consensus.cc:504] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } }
I20260619 11:09:58.752549 11666 leader_election.cc:302] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f752330fa73a4ecab320dd0ad5a32238; no voters:
I20260619 11:09:58.752888 11666 leader_election.cc:288] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:58.753002 11671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:58.753242 11671 raft_consensus.cc:686] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [term 1 LEADER]: Becoming Leader. State: Replica: f752330fa73a4ecab320dd0ad5a32238, State: Running, Role: LEADER
I20260619 11:09:58.753710 11671 consensus_queue.cc:237] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } }
I20260619 11:09:58.754109 11666 sys_catalog.cc:564] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:58.755335 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f752330fa73a4ecab320dd0ad5a32238. Latest consensus state: current_term: 1 leader_uuid: "f752330fa73a4ecab320dd0ad5a32238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } } }
I20260619 11:09:58.755461 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:58.755769 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f752330fa73a4ecab320dd0ad5a32238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f752330fa73a4ecab320dd0ad5a32238" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37323 } } }
I20260619 11:09:58.755839 11680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:58.756321 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:58.758737 11680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:58.763720 11680 catalog_manager.cc:1347] Generated new cluster ID: b1bb6862172049158613794b8a4559e2
I20260619 11:09:58.763792 11680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:58.769649 11680 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:58.770457 11680 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:58.777238 11680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: Generated new TSK 0
I20260619 11:09:58.778168 11680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:09:58.807915 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:58.919064 11692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:58.919313 11692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:58.919376 11692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:58.923305 11692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:58.923363 11692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:58.923429 11692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:09:58.929205 11692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:58.930274 11692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:58.931249 11692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:58.940696 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:58.940809 11692 server_base.cc:1034] running on GCE node
W20260619 11:09:58.940688 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:58.940734 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:58.941427 11692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:58.942080 11692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:58.943279 11692 hybrid_clock.cc:648] HybridClock initialized: now 1781867398943256 us; error 29 us; skew 500 ppm
Jun 19 11:09:58 dist-test-slave-f1t5 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867398, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:58.945930 11692 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:09:58.946550 11692 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:58.946617 11692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:58.946768 11692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:58.948668 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "fb7dfb76e387441bba82f2e8f171ab9c"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "850cd8dc1389e22e6b67cc866708330e"
server_key_iv: "5235332a66cedcf4465867a39475ebc6"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.949144 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "fb7dfb76e387441bba82f2e8f171ab9c"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "850cd8dc1389e22e6b67cc866708330e"
server_key_iv: "5235332a66cedcf4465867a39475ebc6"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.952418 11692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:58.954656 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:58.955888 11692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:58.955981 11692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "fb7dfb76e387441bba82f2e8f171ab9c"
format_stamp: "Formatted at 2026-06-19 11:09:58 on dist-test-slave-f1t5"
server_key: "850cd8dc1389e22e6b67cc866708330e"
server_key_iv: "5235332a66cedcf4465867a39475ebc6"
server_key_version: "encryptionkey@0"
I20260619 11:09:58.956054 11692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:58.964465 11692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:58.967178 11692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:58.967322 11692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:58.967767 11692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:58.968489 11692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:58.968532 11692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:58.968567 11692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:58.968597 11692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:58.977180 11692 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:37787
I20260619 11:09:58.977257 11819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:37787 every 8 connection(s)
I20260619 11:09:58.977789 11692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:58.983974 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11692
I20260619 11:09:58.984232 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:58.984555 7529 external_mini_cluster.cc:1428] Setting key af26f2f639a3c804414de6ac4d221924
Jun 19 11:09:58 dist-test-slave-f1t5 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867398, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:58.991489 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:58.979499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:43589 (local address 127.7.90.126:37323)
0619 11:09:58.979641 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:58.979644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:58.980434 (+ 790us) server_negotiation.cc:407] Connection header received
0619 11:09:58.981712 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:58.981716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:58.981769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:58.981872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:58.983006 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.983533 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.984888 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.985125 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.985236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:58.987878 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:58.987884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:58.987886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:58.987907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:58.989285 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:58.989859 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:58.989864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:58.989867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:58.989973 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:58.990391 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:58.990393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:58.990395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:58.990555 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:58.990631 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:58.991280 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 11:09:58.991374 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:09:58.992726 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:58.979853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37323 (local address 127.7.90.65:43589)
0619 11:09:58.980292 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:58.980332 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:58.981457 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:58.982037 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:58.982044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:58.982398 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:58.982866 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:58.982878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.983665 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.983669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:58.984770 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:58.984779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:58.985268 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:58.985272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:58.985435 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:58.986076 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:58.986094 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:58.987716 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:58.989404 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:58.989410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:58.989423 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:58.989750 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:58.990094 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:58.990101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:58.990104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:58.990291 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:58.990671 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:58.990680 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:58.991020 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 11:09:58.991241 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 11:09:58.991604 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260619 11:09:58.994081 11820 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37323
I20260619 11:09:58.994318 11820 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:09:58.994784 11820 heartbeater.cc:511] Master 127.7.90.126:37323 requested a full tablet report, sending...
I20260619 11:09:58.996222 11630 ts_manager.cc:194] Registered new tserver with Master: fb7dfb76e387441bba82f2e8f171ab9c (127.7.90.65:37787)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:09:58.997323 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:43589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:09:59.010445 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:59.117889 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:59.118153 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:59.118197 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:59.122233 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:59.122285 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:59.122344 11825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:09:59.127451 11825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:59.128484 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:59.129487 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:09:59.138742 11825 server_base.cc:1034] running on GCE node
W20260619 11:09:59.138648 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:59.138710 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:59.138636 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:59.139382 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:59.139936 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:59.141089 11825 hybrid_clock.cc:648] HybridClock initialized: now 1781867399141075 us; error 25 us; skew 500 ppm
Jun 19 11:09:59 dist-test-slave-f1t5 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:59.143707 11825 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:09:59.144392 11825 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:59.144481 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:59.144709 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:59.146759 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "5d5c8a9c6a4b408fafb542cbb24cb118"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cacc852f4c17322e6636c31a75afc6da"
server_key_iv: "7b9a40ead42b0485a457d86eb122b1cf"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.147293 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "5d5c8a9c6a4b408fafb542cbb24cb118"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cacc852f4c17322e6636c31a75afc6da"
server_key_iv: "7b9a40ead42b0485a457d86eb122b1cf"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.150758 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:59.153016 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.154263 11825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:59.154392 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "5d5c8a9c6a4b408fafb542cbb24cb118"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cacc852f4c17322e6636c31a75afc6da"
server_key_iv: "7b9a40ead42b0485a457d86eb122b1cf"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.154498 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:59.170799 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:59.173583 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:59.173797 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:59.174350 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:59.175150 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:59.175231 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.175321 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:59.175371 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.183773 11825 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:39695
I20260619 11:09:59.183794 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:39695 every 8 connection(s)
I20260619 11:09:59.184327 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:59.186682 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11825
I20260619 11:09:59.186798 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:59.187098 7529 external_mini_cluster.cc:1428] Setting key e0e6af05663d18044c1ce9305f85ecf0
Jun 19 11:09:59 dist-test-slave-f1t5 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:09:59.197659 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.185998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:39177 (local address 127.7.90.126:37323)
0619 11:09:59.186133 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.186136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.187064 (+ 928us) server_negotiation.cc:407] Connection header received
0619 11:09:59.188509 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.188512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.188562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.188640 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:59.189752 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.190257 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.191334 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.191548 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.191650 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.194064 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:59.194067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:59.194069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:59.194091 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:59.195517 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.196076 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.196081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.196083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.196164 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.196497 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.196500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.196502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.196743 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:59.196851 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.197338 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.197475 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:09:59.198607 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.186369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37323 (local address 127.7.90.66:39177)
0619 11:09:59.186933 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.186975 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.188290 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.188799 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.188806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.189151 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:59.189612 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.189625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.190373 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.190378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.191213 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.191221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.191681 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.191685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.191872 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.192450 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:59.192474 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:59.193883 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:59.195631 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:59.195637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:59.195651 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:59.195979 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:59.196261 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:59.196264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:59.196266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:59.196399 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:59.196858 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:59.196866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:59.197113 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.197327 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.197703 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260619 11:09:59.199731 11953 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37323
I20260619 11:09:59.199990 11953 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:59.200440 11953 heartbeater.cc:511] Master 127.7.90.126:37323 requested a full tablet report, sending...
I20260619 11:09:59.201488 11630 ts_manager.cc:194] Registered new tserver with Master: 5d5c8a9c6a4b408fafb542cbb24cb118 (127.7.90.66:39695)
I20260619 11:09:59.202390 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:39177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:09:59.212549 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:59.320544 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:59.320770 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:59.320811 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:59.324554 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:59.324600 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:59.324654 11958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:09:59.329774 11958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:59.330662 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:59.331591 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:59.340797 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:59.340862 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:59.340873 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:59.341014 11958 server_base.cc:1034] running on GCE node
I20260619 11:09:59.341667 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:59.342286 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:59.343582 11958 hybrid_clock.cc:648] HybridClock initialized: now 1781867399343549 us; error 53 us; skew 500 ppm
Jun 19 11:09:59 dist-test-slave-f1t5 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:59.346297 11958 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:09:59.346983 11958 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:59.347071 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:59.347291 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:59.349283 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "36ea511cf9c842e6bf88ad7fc84a890d"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cd1f8e0c6c1258b3c9cebfd5749189c8"
server_key_iv: "3dd268bf524361f19a0b678bb5c65134"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.349866 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "36ea511cf9c842e6bf88ad7fc84a890d"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cd1f8e0c6c1258b3c9cebfd5749189c8"
server_key_iv: "3dd268bf524361f19a0b678bb5c65134"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.353397 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:09:59.355867 11972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.357016 11958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:59.357142 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "36ea511cf9c842e6bf88ad7fc84a890d"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "cd1f8e0c6c1258b3c9cebfd5749189c8"
server_key_iv: "3dd268bf524361f19a0b678bb5c65134"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.357245 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:59.376447 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:59.379320 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:59.379515 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:59.380088 11958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:59.380923 11958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:59.380999 11958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.381074 11958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:59.381129 11958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.389431 11958 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:34725
I20260619 11:09:59.389451 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:34725 every 8 connection(s)
I20260619 11:09:59.390075 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:09:59 dist-test-slave-f1t5 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:09:59.399048 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 11958
I20260619 11:09:59.399139 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:59.399420 7529 external_mini_cluster.cc:1428] Setting key e735a42646387299e3e495ff5ebba3e2
I20260619 11:09:59.402928 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.391745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:39159 (local address 127.7.90.126:37323)
0619 11:09:59.391887 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.391891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.392752 (+ 861us) server_negotiation.cc:407] Connection header received
0619 11:09:59.394003 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.394006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.394051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.394122 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:59.395194 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.395697 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.396771 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.396980 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.397052 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.399488 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:59.399495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:59.399498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:59.399524 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:59.401064 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.401659 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.401662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.401664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.401737 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.402040 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.402043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.402044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.402222 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:59.402296 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.402726 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.402817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:09:59.403697 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.392166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37323 (local address 127.7.90.67:39159)
0619 11:09:59.392588 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.392628 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.393818 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.394273 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.394279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.394616 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:59.395063 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.395086 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.395798 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.395800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.396670 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.396679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.397102 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.397104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.397226 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.397839 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:59.397854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:59.399332 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:59.401191 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:59.401196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:59.401206 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:59.401551 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:59.401815 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:59.401817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:59.401818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:59.401941 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:59.402310 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:59.402314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:59.402528 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.402667 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.402888 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260619 11:09:59.404651 12086 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37323
I20260619 11:09:59.404865 12086 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:59.405329 12086 heartbeater.cc:511] Master 127.7.90.126:37323 requested a full tablet report, sending...
I20260619 11:09:59.406373 11630 ts_manager.cc:194] Registered new tserver with Master: 36ea511cf9c842e6bf88ad7fc84a890d (127.7.90.67:34725)
I20260619 11:09:59.406873 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:39159
I20260619 11:09:59.412943 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:09:59.421087 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.414783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.7.90.126:37323)
0619 11:09:59.414931 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.414937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.415044 (+ 107us) server_negotiation.cc:407] Connection header received
0619 11:09:59.415133 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.415135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.415171 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.415226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:59.415791 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.416289 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.417008 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.417202 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.417272 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.418539 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:59.418541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:59.418543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:59.418561 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:59.419715 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.420062 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.420064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.420065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.420139 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:59.420347 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:59.420349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:59.420350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:59.420530 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:59.420626 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.420802 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.420882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:09:59.423805 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:59.427270 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:59.441555 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.436126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.7.90.67:34725)
0619 11:09:59.436456 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.436467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.436727 (+ 260us) server_negotiation.cc:407] Connection header received
0619 11:09:59.436790 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.436793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.436934 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.437094 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:59.437371 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.438403 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.440252 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.440866 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.440992 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441137 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.441297 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.441385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":132,"threads_started":1}
I20260619 11:09:59.441817 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.435328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:34725 (local address 127.0.0.1:38138)
0619 11:09:59.436517 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.436534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.436630 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.437098 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.437104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.437124 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:59.437249 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.437255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.438550 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.438554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.440132 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.440138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.440990 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.440994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.441106 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441140 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.441252 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.441373 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":133,"thread_start_us":83,"threads_started":1}
I20260619 11:09:59.442090 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.436036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:39695 (local address 127.0.0.1:37120)
0619 11:09:59.437240 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.437250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.437356 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.437817 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.437820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.437842 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:59.437928 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.437934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.439415 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.439420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.440310 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.440322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.441452 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.441455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.441603 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441837 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.441910 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.441975 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":64,"thread_start_us":76,"threads_started":1}
I20260619 11:09:59.442135 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.435605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.7.90.65:37787)
0619 11:09:59.436065 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.436094 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.436658 (+ 564us) server_negotiation.cc:407] Connection header received
0619 11:09:59.436742 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.436748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.436935 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.437052 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:59.437497 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.438673 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.440265 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.441063 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.441166 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441298 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.441884 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.441991 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":146,"threads_started":1}
I20260619 11:09:59.442531 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.436402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.7.90.66:39695)
0619 11:09:59.436744 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.436755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.437341 (+ 586us) server_negotiation.cc:407] Connection header received
0619 11:09:59.437464 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.437471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.437675 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.437822 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:59.438056 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.439280 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.440438 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.441203 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.441291 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441403 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.442308 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.442413 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":131,"threads_started":1}
I20260619 11:09:59.441834 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.435315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37787 (local address 127.0.0.1:37802)
0619 11:09:59.436330 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.436379 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.436514 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.437245 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.437248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.437262 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:59.437359 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.437365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.438797 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.438800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.440113 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.440124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.441359 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.441363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.441455 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.441477 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.441607 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.441685 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":68,"thread_start_us":68,"threads_started":1}
I20260619 11:09:59.445098 11754 tablet_service.cc:1467] Processing CreateTablet for tablet 10440c4052c7450e8bcf672ca8747b54 (DEFAULT_TABLE table=test-table [id=396b570e419e41c28dd55de505980ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:59.445128 12020 tablet_service.cc:1467] Processing CreateTablet for tablet 10440c4052c7450e8bcf672ca8747b54 (DEFAULT_TABLE table=test-table [id=396b570e419e41c28dd55de505980ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:59.445181 11887 tablet_service.cc:1467] Processing CreateTablet for tablet 10440c4052c7450e8bcf672ca8747b54 (DEFAULT_TABLE table=test-table [id=396b570e419e41c28dd55de505980ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:59.446098 11754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10440c4052c7450e8bcf672ca8747b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:59.446141 12020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10440c4052c7450e8bcf672ca8747b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:59.446131 11887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10440c4052c7450e8bcf672ca8747b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:59.451762 12103 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Bootstrap starting.
I20260619 11:09:59.453364 12105 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Bootstrap starting.
I20260619 11:09:59.454047 12103 tablet_bootstrap.cc:654] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:59.454820 12104 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Bootstrap starting.
I20260619 11:09:59.455413 12103 log.cc:824] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:59.456642 12105 tablet_bootstrap.cc:654] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:59.456719 12104 tablet_bootstrap.cc:654] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:59.457605 12103 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: No bootstrap required, opened a new log
I20260619 11:09:59.457799 12103 ts_tablet_manager.cc:1397] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:59.457862 12105 log.cc:824] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:59.457938 12104 log.cc:824] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:59.459731 12105 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: No bootstrap required, opened a new log
I20260619 11:09:59.459926 12105 ts_tablet_manager.cc:1397] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 11:09:59.460507 12104 tablet_bootstrap.cc:492] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: No bootstrap required, opened a new log
I20260619 11:09:59.460673 12104 ts_tablet_manager.cc:1397] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:59.461679 12103 raft_consensus.cc:348] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.461946 12103 raft_consensus.cc:374] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:59.461997 12103 raft_consensus.cc:729] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7dfb76e387441bba82f2e8f171ab9c, State: Initialized, Role: FOLLOWER
I20260619 11:09:59.462424 12103 consensus_queue.cc:260] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.463060 11820 heartbeater.cc:503] Master 127.7.90.126:37323 was elected leader, sending a full tablet report...
I20260619 11:09:59.463160 12105 raft_consensus.cc:348] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.463366 12103 ts_tablet_manager.cc:1428] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:09:59.463375 12105 raft_consensus.cc:374] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:59.463418 12105 raft_consensus.cc:729] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5c8a9c6a4b408fafb542cbb24cb118, State: Initialized, Role: FOLLOWER
I20260619 11:09:59.463797 12105 consensus_queue.cc:260] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.464150 12104 raft_consensus.cc:348] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.464387 12104 raft_consensus.cc:374] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:59.464433 12104 raft_consensus.cc:729] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ea511cf9c842e6bf88ad7fc84a890d, State: Initialized, Role: FOLLOWER
I20260619 11:09:59.464695 12105 ts_tablet_manager.cc:1428] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:59.464797 12104 consensus_queue.cc:260] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.465381 12086 heartbeater.cc:503] Master 127.7.90.126:37323 was elected leader, sending a full tablet report...
I20260619 11:09:59.465664 11953 heartbeater.cc:503] Master 127.7.90.126:37323 was elected leader, sending a full tablet report...
I20260619 11:09:59.465691 12104 ts_tablet_manager.cc:1428] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:09:59.478722 11821 tablet.cc:2369] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:09:59.641124 12087 tablet.cc:2369] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:09:59.685809 11954 tablet.cc:2369] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:59.687503 12110 raft_consensus.cc:482] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:59.687705 12110 raft_consensus.cc:504] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.688791 12110 leader_election.cc:288] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb7dfb76e387441bba82f2e8f171ab9c (127.7.90.65:37787), 36ea511cf9c842e6bf88ad7fc84a890d (127.7.90.67:34725)
I20260619 11:09:59.693657 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.689177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:41983 (local address 127.7.90.65:37787)
0619 11:09:59.689307 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.689313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.689363 (+ 50us) server_negotiation.cc:407] Connection header received
0619 11:09:59.689488 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.689491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.689580 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.689666 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:59.689944 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.690718 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.692405 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.693024 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.693137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.693201 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.693430 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.693498 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:09:59.693657 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.689177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:45193 (local address 127.7.90.67:34725)
0619 11:09:59.689307 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.689312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.689558 (+ 246us) server_negotiation.cc:407] Connection header received
0619 11:09:59.689681 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.689684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.689728 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.689803 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:59.690049 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.690733 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.692405 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.693024 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.693138 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.693201 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.693430 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.693498 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:09:59.693749 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.689076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37787 (local address 127.7.90.66:41983)
0619 11:09:59.689274 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.689291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.689375 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.689692 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.689699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.689711 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:59.689834 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.689840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.690830 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.690833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.692288 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.692298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.693148 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.693151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.693315 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.693331 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.693400 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.693559 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":179}
I20260619 11:09:59.693758 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.689076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:34725 (local address 127.7.90.66:45193)
0619 11:09:59.689446 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.689459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:59.689586 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:59.689825 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:59.689827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:59.689839 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:59.689947 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.689953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.690830 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.690833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.692288 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:59.692298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.693148 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.693151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:59.693285 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.693322 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 11:09:59.693400 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:09:59.693560 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":255,"thread_start_us":92,"threads_started":1}
I20260619 11:09:59.694500 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10440c4052c7450e8bcf672ca8747b54" candidate_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" is_pre_election: true
I20260619 11:09:59.694500 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10440c4052c7450e8bcf672ca8747b54" candidate_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" is_pre_election: true
I20260619 11:09:59.694805 12040 raft_consensus.cc:2454] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5c8a9c6a4b408fafb542cbb24cb118 in term 0.
I20260619 11:09:59.694806 11774 raft_consensus.cc:2454] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d5c8a9c6a4b408fafb542cbb24cb118 in term 0.
I20260619 11:09:59.695315 11843 leader_election.cc:302] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36ea511cf9c842e6bf88ad7fc84a890d, 5d5c8a9c6a4b408fafb542cbb24cb118, fb7dfb76e387441bba82f2e8f171ab9c; no voters:
I20260619 11:09:59.695590 12110 raft_consensus.cc:2781] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:59.695686 12110 raft_consensus.cc:482] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:59.695747 12110 raft_consensus.cc:3037] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:59.696734 12110 raft_consensus.cc:504] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.697062 12110 leader_election.cc:288] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [CANDIDATE]: Term 1 election: Requested vote from peers fb7dfb76e387441bba82f2e8f171ab9c (127.7.90.65:37787), 36ea511cf9c842e6bf88ad7fc84a890d (127.7.90.67:34725)
I20260619 11:09:59.697402 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10440c4052c7450e8bcf672ca8747b54" candidate_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb7dfb76e387441bba82f2e8f171ab9c"
I20260619 11:09:59.697502 11774 raft_consensus.cc:3037] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:59.697656 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10440c4052c7450e8bcf672ca8747b54" candidate_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ea511cf9c842e6bf88ad7fc84a890d"
I20260619 11:09:59.697778 12040 raft_consensus.cc:3037] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:59.698616 11774 raft_consensus.cc:2454] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5c8a9c6a4b408fafb542cbb24cb118 in term 1.
I20260619 11:09:59.698861 12040 raft_consensus.cc:2454] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d5c8a9c6a4b408fafb542cbb24cb118 in term 1.
I20260619 11:09:59.698904 11840 leader_election.cc:302] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d5c8a9c6a4b408fafb542cbb24cb118, fb7dfb76e387441bba82f2e8f171ab9c; no voters:
I20260619 11:09:59.699086 12110 raft_consensus.cc:2781] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:59.699316 12110 raft_consensus.cc:686] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 1 LEADER]: Becoming Leader. State: Replica: 5d5c8a9c6a4b408fafb542cbb24cb118, State: Running, Role: LEADER
I20260619 11:09:59.699621 12110 consensus_queue.cc:237] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } }
I20260619 11:09:59.702600 11628 catalog_manager.cc:5495] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d5c8a9c6a4b408fafb542cbb24cb118 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d5c8a9c6a4b408fafb542cbb24cb118" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39695 } health_report { overall_health: HEALTHY } } }
I20260619 11:09:59.734923 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.731246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.7.90.66:39695)
0619 11:09:59.731369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.731372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.731404 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:09:59.731469 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.731471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.731520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.731609 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:59.731872 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.732382 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.733505 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:59.733738 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:59.733812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:59.734118 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:59.734484 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:59.734569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.734681 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.734726 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:09:59.741082 11774 raft_consensus.cc:1261] T 10440c4052c7450e8bcf672ca8747b54 P fb7dfb76e387441bba82f2e8f171ab9c [term 1 FOLLOWER]: Refusing update from remote peer 5d5c8a9c6a4b408fafb542cbb24cb118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:59.741082 12040 raft_consensus.cc:1261] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 1 FOLLOWER]: Refusing update from remote peer 5d5c8a9c6a4b408fafb542cbb24cb118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:59.741775 12113 consensus_queue.cc:1035] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ea511cf9c842e6bf88ad7fc84a890d" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:59.741947 12110 consensus_queue.cc:1035] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb7dfb76e387441bba82f2e8f171ab9c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:59.749620 12116 mvcc.cc:205] Tried to move back new op lower bound from 7298528869332205568 to 7298528869172236288. Current Snapshot: MvccSnapshot[applied={T|T < 7298528869332205568}]
I20260619 11:09:59.751539 12118 mvcc.cc:205] Tried to move back new op lower bound from 7298528869332205568 to 7298528869172236288. Current Snapshot: MvccSnapshot[applied={T|T < 7298528869332205568}]
I20260619 11:09:59.758189 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:59.758317 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:59.760702 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f752330fa73a4ecab320dd0ad5a32238: Sending DeleteTablet for 3 replicas of tablet 10440c4052c7450e8bcf672ca8747b54
I20260619 11:09:59.761412 11754 tablet_service.cc:1514] Processing DeleteTablet for tablet 10440c4052c7450e8bcf672ca8747b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:59 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:37802
I20260619 11:09:59.761409 11887 tablet_service.cc:1514] Processing DeleteTablet for tablet 10440c4052c7450e8bcf672ca8747b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:59 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:37120
I20260619 11:09:59.761447 12020 tablet_service.cc:1514] Processing DeleteTablet for tablet 10440c4052c7450e8bcf672ca8747b54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:59 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:38138
I20260619 11:09:59.761981 12125 tablet_replica.cc:330] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: stopping tablet replica
I20260619 11:09:59.762290 12125 raft_consensus.cc:2229] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:59.762492 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11692
I20260619 11:09:59.764187 12125 raft_consensus.cc:2258] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:59.765585 12125 ts_tablet_manager.cc:1905] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:59.765735 12127 tablet_replica.cc:330] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: stopping tablet replica
I20260619 11:09:59.766036 12127 raft_consensus.cc:2229] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:59.766345 12127 raft_consensus.cc:2258] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:59.767706 12127 ts_tablet_manager.cc:1905] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:59.769704 12125 ts_tablet_manager.cc:1918] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:59.769856 12125 log.cc:1196] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/10440c4052c7450e8bcf672ca8747b54
I20260619 11:09:59.770255 12125 ts_tablet_manager.cc:1939] T 10440c4052c7450e8bcf672ca8747b54 P 5d5c8a9c6a4b408fafb542cbb24cb118: Deleting consensus metadata
I20260619 11:09:59.770277 12127 ts_tablet_manager.cc:1918] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:59.770440 12127 log.cc:1196] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/10440c4052c7450e8bcf672ca8747b54
I20260619 11:09:59.770816 12127 ts_tablet_manager.cc:1939] T 10440c4052c7450e8bcf672ca8747b54 P 36ea511cf9c842e6bf88ad7fc84a890d: Deleting consensus metadata
W20260619 11:09:59.770910 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:37787 (error 108)
I20260619 11:09:59.771457 11614 catalog_manager.cc:4841] TS 5d5c8a9c6a4b408fafb542cbb24cb118 (127.7.90.66:39695): tablet 10440c4052c7450e8bcf672ca8747b54 (table test-table [id=396b570e419e41c28dd55de505980ae8]) successfully deleted
I20260619 11:09:59.771901 11617 catalog_manager.cc:4841] TS 36ea511cf9c842e6bf88ad7fc84a890d (127.7.90.67:34725): tablet 10440c4052c7450e8bcf672ca8747b54 (table test-table [id=396b570e419e41c28dd55de505980ae8]) successfully deleted
I20260619 11:09:59.772190 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11825
I20260619 11:09:59.772482 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.772197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37787 (local address 127.0.0.1:37808)
0619 11:09:59.772321 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:59.772391 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 11:09:59.774072 11614 catalog_manager.cc:4584] TS fb7dfb76e387441bba82f2e8f171ab9c (127.7.90.65:37787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10440c4052c7450e8bcf672ca8747b54: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37787: connect: Connection refused (error 111)
I20260619 11:09:59.777796 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11958
I20260619 11:09:59.784041 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 11599
2026-06-19T11:09:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2374 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T11:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:59Z Disabled control of system clock
I20260619 11:09:59.814126 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46717
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:46717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:09:59.921738 12132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:59.921948 12132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:59.921979 12132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:59.925799 12132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:59.925843 12132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:59.925858 12132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:59.925873 12132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:59.925887 12132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:59.930342 12132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:46717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:59.931244 12132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:59.932442 12132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:59.940263 12140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:59.940271 12138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:59.940394 12132 server_base.cc:1034] running on GCE node
W20260619 11:09:59.940582 12137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:59.941114 12132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:59.942149 12132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:59.943305 12132 hybrid_clock.cc:648] HybridClock initialized: now 1781867399943286 us; error 29 us; skew 500 ppm
I20260619 11:09:59.944994 12132 webserver.cc:460] Webserver started at http://127.7.90.126:38249/ using document root <none> and password file <none>
I20260619 11:09:59.945581 12132 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:59.945642 12132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:59.945858 12132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:59.947753 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "d779c210b0a645e398a56b82cf6735b4"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "c23f8d8ae23e61525408dcab24749a62"
server_key_iv: "96d68e3522c97ba106d375da2b30e30d"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.948294 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "d779c210b0a645e398a56b82cf6735b4"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "c23f8d8ae23e61525408dcab24749a62"
server_key_iv: "96d68e3522c97ba106d375da2b30e30d"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.951794 12132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:09:59.954012 12146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:59.955135 12132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:59.955267 12132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "d779c210b0a645e398a56b82cf6735b4"
format_stamp: "Formatted at 2026-06-19 11:09:59 on dist-test-slave-f1t5"
server_key: "c23f8d8ae23e61525408dcab24749a62"
server_key_iv: "96d68e3522c97ba106d375da2b30e30d"
server_key_version: "encryptionkey@0"
I20260619 11:09:59.955370 12132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:59.959620 12132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:59.960166 12132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:59.960335 12132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:59.966500 12132 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:46717
I20260619 11:09:59.966506 12198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:46717 every 8 connection(s)
I20260619 11:09:59.967334 12132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:59.968896 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12132
I20260619 11:09:59.969033 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:09:59.969300 7529 external_mini_cluster.cc:1428] Setting key e815a7a0c8144b787e22f6810e5eb048
I20260619 11:09:59.970575 12199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:59.974614 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:59.970776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.7.90.126:46717)
0619 11:09:59.971365 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:59.971379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:59.971412 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:59.972024 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:59.972039 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:59.972355 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:59.972681 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:59.973405 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:59.973417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:59.973428 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:59.973438 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:59.973491 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:59.973507 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:59.973683 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:59.973764 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 11:09:59.973993 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260619 11:09:59.976356 12199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: Bootstrap starting.
I20260619 11:09:59.978506 12199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:59.979249 12199 log.cc:824] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:59.980983 12199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: No bootstrap required, opened a new log
I20260619 11:09:59.984175 12199 raft_consensus.cc:348] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } }
I20260619 11:09:59.984392 12199 raft_consensus.cc:374] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:59.984426 12199 raft_consensus.cc:729] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d779c210b0a645e398a56b82cf6735b4, State: Initialized, Role: FOLLOWER
I20260619 11:09:59.984748 12199 consensus_queue.cc:260] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } }
I20260619 11:09:59.984848 12199 raft_consensus.cc:388] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:59.984930 12199 raft_consensus.cc:482] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:59.985034 12199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:59.986058 12199 raft_consensus.cc:504] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } }
I20260619 11:09:59.986423 12199 leader_election.cc:302] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d779c210b0a645e398a56b82cf6735b4; no voters:
I20260619 11:09:59.986722 12199 leader_election.cc:288] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:59.986832 12204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:59.987102 12204 raft_consensus.cc:686] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [term 1 LEADER]: Becoming Leader. State: Replica: d779c210b0a645e398a56b82cf6735b4, State: Running, Role: LEADER
I20260619 11:09:59.987478 12204 consensus_queue.cc:237] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } }
I20260619 11:09:59.987902 12199 sys_catalog.cc:564] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:59.989038 12205 sys_catalog.cc:455] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d779c210b0a645e398a56b82cf6735b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } } }
I20260619 11:09:59.989171 12205 sys_catalog.cc:458] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:59.989567 12213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:59.989537 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d779c210b0a645e398a56b82cf6735b4. Latest consensus state: current_term: 1 leader_uuid: "d779c210b0a645e398a56b82cf6735b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d779c210b0a645e398a56b82cf6735b4" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46717 } } }
I20260619 11:09:59.989795 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:59.992164 12213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:59.996989 12213 catalog_manager.cc:1347] Generated new cluster ID: 0bcb2a7aeb454c0da09efc64703ca8a3
I20260619 11:09:59.997047 12213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:00.019423 12213 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:00.019791 12213 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:00.027416 12213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: Generated new TSK 0
I20260619 11:10:00.028273 12213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:00.044817 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--builtin_ntp_servers=127.7.90.84:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:10:00.153859 12223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:00.154089 12223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:00.154129 12223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:00.158073 12223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:00.158124 12223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:00.158207 12223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:00.162978 12223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:00.164021 12223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:00.165364 12223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:00.174664 12229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.174682 12231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.174646 12228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.174834 12223 server_base.cc:1034] running on GCE node
I20260619 11:10:00.175333 12223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:00.175902 12223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:00.177093 12223 hybrid_clock.cc:648] HybridClock initialized: now 1781867400177077 us; error 26 us; skew 500 ppm
I20260619 11:10:00.178902 12223 webserver.cc:460] Webserver started at http://127.7.90.65:35745/ using document root <none> and password file <none>
I20260619 11:10:00.179471 12223 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:00.179560 12223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:00.179774 12223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:00.181833 12223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "2d5a0cf914a042f4b88b539a189fe434"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "e13fbd3cb0805f15e92be4bdb0e4b515"
server_key_iv: "6cdd94889996e861d6f683423ad6dae9"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.182379 12223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5a0cf914a042f4b88b539a189fe434"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "e13fbd3cb0805f15e92be4bdb0e4b515"
server_key_iv: "6cdd94889996e861d6f683423ad6dae9"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.185873 12223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:00.188374 12237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.189772 12223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:00.189914 12223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "2d5a0cf914a042f4b88b539a189fe434"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "e13fbd3cb0805f15e92be4bdb0e4b515"
server_key_iv: "6cdd94889996e861d6f683423ad6dae9"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.190030 12223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:00.194542 12223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:00.195120 12223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:00.195297 12223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:00.195806 12223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:00.196679 12223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:00.196750 12223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.196812 12223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:00.196866 12223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.205699 12223 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:44521
I20260619 11:10:00.205718 12350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:44521 every 8 connection(s)
I20260619 11:10:00.206549 12223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:00.210999 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12223
I20260619 11:10:00.211113 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:00.211434 7529 external_mini_cluster.cc:1428] Setting key cb1597169aaa753fc301ce979ace9f3f
I20260619 11:10:00.217001 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--builtin_ntp_servers=127.7.90.84:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 11:10:00.218957 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.208364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:46029 (local address 127.7.90.126:46717)
0619 11:10:00.208546 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.208552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.215145 (+ 6593us) server_negotiation.cc:407] Connection header received
0619 11:10:00.216967 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.216970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.217005 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.217062 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.218391 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.218396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.218399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.218416 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.218445 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.218457 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.218564 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.218739 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.218790 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 11:10:00.219897 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.208845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46717 (local address 127.7.90.65:46029)
0619 11:10:00.214972 (+ 6127us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.215036 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.216722 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.217251 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.217264 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.217385 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.218030 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.218035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.218102 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.218198 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.218567 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.218578 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.218584 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.218747 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.218948 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5586,"thread_start_us":149,"threads_started":1}
I20260619 11:10:00.221089 12351 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46717
I20260619 11:10:00.221364 12351 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:00.221943 12351 heartbeater.cc:511] Master 127.7.90.126:46717 requested a full tablet report, sending...
I20260619 11:10:00.223498 12163 ts_manager.cc:194] Registered new tserver with Master: 2d5a0cf914a042f4b88b539a189fe434 (127.7.90.65:44521)
W20260619 11:10:00.327771 12354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:00.328086 12354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:00.328176 12354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:00.332123 12354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:00.332211 12354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:00.332325 12354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:00.337052 12354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:00.338135 12354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:00.339471 12354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:00.349191 12362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.349226 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.349288 12354 server_base.cc:1034] running on GCE node
W20260619 11:10:00.349233 12359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.349855 12354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:00.350457 12354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:00.351646 12354 hybrid_clock.cc:648] HybridClock initialized: now 1781867400351617 us; error 47 us; skew 500 ppm
I20260619 11:10:00.353384 12354 webserver.cc:460] Webserver started at http://127.7.90.66:41543/ using document root <none> and password file <none>
I20260619 11:10:00.354005 12354 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:00.354092 12354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:00.354316 12354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:00.356269 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "376c1592e62743e2a34add4e24cffd93"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "838caebd130881d0503da412cd09a3c9"
server_key_iv: "ca6167129c9bd0cbd868a7396361a68c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.356801 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "376c1592e62743e2a34add4e24cffd93"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "838caebd130881d0503da412cd09a3c9"
server_key_iv: "ca6167129c9bd0cbd868a7396361a68c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.360392 12354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:10:00.362643 12368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.363866 12354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:00.364002 12354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "376c1592e62743e2a34add4e24cffd93"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "838caebd130881d0503da412cd09a3c9"
server_key_iv: "ca6167129c9bd0cbd868a7396361a68c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.364110 12354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:00.368566 12354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:00.369139 12354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:00.369313 12354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:00.369897 12354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:00.370764 12354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:00.370833 12354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.370896 12354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:00.370939 12354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.379279 12354 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:42239
I20260619 11:10:00.379302 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:42239 every 8 connection(s)
I20260619 11:10:00.380138 12354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:00.383654 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12354
I20260619 11:10:00.383744 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:00.384004 7529 external_mini_cluster.cc:1428] Setting key a9a684973922abfa7a178e38e72389e3
I20260619 11:10:00.386221 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--builtin_ntp_servers=127.7.90.84:46351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 11:10:00.386257 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.381955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:44863 (local address 127.7.90.126:46717)
0619 11:10:00.382101 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.382104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.383088 (+ 984us) server_negotiation.cc:407] Connection header received
0619 11:10:00.384435 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.384438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.384478 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.384591 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.385789 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.385793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.385796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.385808 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.385833 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.385843 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.385936 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.386071 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.386117 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:10:00.387239 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.382300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46717 (local address 127.7.90.66:44863)
0619 11:10:00.382928 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.382983 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.384208 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.384771 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.384780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.384886 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.385507 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.385509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.385586 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.385651 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.385928 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.385933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.385937 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.386075 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.386253 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260619 11:10:00.388330 12482 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46717
I20260619 11:10:00.388584 12482 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:00.389066 12482 heartbeater.cc:511] Master 127.7.90.126:46717 requested a full tablet report, sending...
I20260619 11:10:00.390022 12163 ts_manager.cc:194] Registered new tserver with Master: 376c1592e62743e2a34add4e24cffd93 (127.7.90.66:42239)
W20260619 11:10:00.497892 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:00.498170 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:00.498239 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:00.502257 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:00.502314 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:00.502409 12485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:00.507277 12485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:00.508395 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:00.509819 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:00.519367 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.519445 12485 server_base.cc:1034] running on GCE node
W20260619 11:10:00.519367 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.519582 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.520026 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:00.520639 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:00.521796 12485 hybrid_clock.cc:648] HybridClock initialized: now 1781867400521762 us; error 43 us; skew 500 ppm
I20260619 11:10:00.523516 12485 webserver.cc:460] Webserver started at http://127.7.90.67:37501/ using document root <none> and password file <none>
I20260619 11:10:00.524078 12485 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:00.524137 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:00.524392 12485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:00.526445 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "f647b42ce2f54200a0f7bf632007a6eb"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "25ed1ee0af91ebf1c6fcbbfec7de29ce"
server_key_iv: "3347271b6e6d647a1d943129d39ecd01"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.527000 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "f647b42ce2f54200a0f7bf632007a6eb"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "25ed1ee0af91ebf1c6fcbbfec7de29ce"
server_key_iv: "3347271b6e6d647a1d943129d39ecd01"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.530649 12485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:10:00.533010 12499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.534734 12485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:00.534868 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "f647b42ce2f54200a0f7bf632007a6eb"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "25ed1ee0af91ebf1c6fcbbfec7de29ce"
server_key_iv: "3347271b6e6d647a1d943129d39ecd01"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.534992 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:00.539902 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:00.540505 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:00.540681 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:00.541215 12485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:00.542152 12485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:00.542223 12485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.542292 12485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:00.542344 12485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.550678 12485 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:43455
I20260619 11:10:00.550698 12612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:43455 every 8 connection(s)
I20260619 11:10:00.551542 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:00.553627 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12485
I20260619 11:10:00.553759 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:00.554077 7529 external_mini_cluster.cc:1428] Setting key 0fc734ca85bbc1dbecd691d4edf403e4
I20260619 11:10:00.557905 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.553243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:50213 (local address 127.7.90.126:46717)
0619 11:10:00.553390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.553395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.554742 (+ 1347us) server_negotiation.cc:407] Connection header received
0619 11:10:00.556306 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.556308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.556342 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.556407 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.557375 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.557378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.557380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.557390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.557408 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.557425 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.557494 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.557728 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.557783 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:10:00.558732 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.553815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46717 (local address 127.7.90.67:50213)
0619 11:10:00.554571 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.554622 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.556113 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.556565 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.556571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.556651 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.557153 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.557155 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.557191 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.557254 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.557571 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.557580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.557586 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.557750 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.557927 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260619 11:10:00.559816 12613 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46717
I20260619 11:10:00.560052 12613 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:00.560539 12613 heartbeater.cc:511] Master 127.7.90.126:46717 requested a full tablet report, sending...
I20260619 11:10:00.561460 12163 ts_manager.cc:194] Registered new tserver with Master: f647b42ce2f54200a0f7bf632007a6eb (127.7.90.67:43455)
I20260619 11:10:00.567636 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:10:00.570827 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.569453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.7.90.126:46717)
0619 11:10:00.569606 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.569609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.569808 (+ 199us) server_negotiation.cc:407] Connection header received
0619 11:10:00.569847 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.569848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.569877 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.569949 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.570491 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.570493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.570495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.570503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.570516 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.570522 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.570595 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.570639 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.570666 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:10:00.573290 12163 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:00.576179 12163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:00.587064 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.584470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.7.90.66:42239)
0619 11:10:00.585131 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.585143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.585160 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:10:00.585203 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.585207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.585348 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.585756 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.586448 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.586454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.586476 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.586488 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.586529 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.586568 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.586674 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.586815 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.586891 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":133,"threads_started":1}
I20260619 11:10:00.587066 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.584117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:43455 (local address 127.0.0.1:33154)
0619 11:10:00.584751 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.584764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.584821 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.585393 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.585398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.585471 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.586385 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.586389 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.586400 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.586425 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.586747 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.586770 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.586775 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.586856 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.586888 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":90,"threads_started":1}
I20260619 11:10:00.587563 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.583853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.7.90.65:44521)
0619 11:10:00.584923 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.584935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.585109 (+ 174us) server_negotiation.cc:407] Connection header received
0619 11:10:00.585167 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.585174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.585349 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.585477 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.586489 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.586494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.586502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.586512 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.586544 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.586569 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.586674 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.587139 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.587206 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":152,"threads_started":1}
I20260619 11:10:00.587605 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.584469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.7.90.67:43455)
0619 11:10:00.584957 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.584969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.584994 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:10:00.585049 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.585054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.585242 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.585393 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.586549 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.586556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.586570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.586580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.586639 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.586670 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.586773 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.587378 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.587443 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":118,"threads_started":1}
I20260619 11:10:00.587080 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.583850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:42239 (local address 127.0.0.1:36500)
0619 11:10:00.584535 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.584584 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.584672 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.585506 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.585509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.585546 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.586201 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.586204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.586230 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.586305 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.586663 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.586672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.586678 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.586789 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.586860 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":106,"threads_started":1}
I20260619 11:10:00.587078 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.583746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44521 (local address 127.0.0.1:39946)
0619 11:10:00.584911 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.584922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.585024 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.585576 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.585580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.585594 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.586310 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.586312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.586320 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.586342 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.586824 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.586826 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.586829 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.586895 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.586923 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":70,"threads_started":1}
I20260619 11:10:00.590310 12416 tablet_service.cc:1467] Processing CreateTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 (DEFAULT_TABLE table=test-table [id=561778e048324ddba5a87e442d6d83c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:00.590371 12547 tablet_service.cc:1467] Processing CreateTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 (DEFAULT_TABLE table=test-table [id=561778e048324ddba5a87e442d6d83c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:00.590633 12285 tablet_service.cc:1467] Processing CreateTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 (DEFAULT_TABLE table=test-table [id=561778e048324ddba5a87e442d6d83c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:00.591298 12416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaeef5a6cc54d5591528d4cfee3ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:00.591361 12547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaeef5a6cc54d5591528d4cfee3ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:00.591403 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaeef5a6cc54d5591528d4cfee3ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:00.597038 12630 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Bootstrap starting.
I20260619 11:10:00.598318 12631 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: Bootstrap starting.
I20260619 11:10:00.599229 12632 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Bootstrap starting.
I20260619 11:10:00.599612 12630 tablet_bootstrap.cc:654] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:00.600312 12631 tablet_bootstrap.cc:654] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:00.600508 12630 log.cc:824] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:00.601049 12632 tablet_bootstrap.cc:654] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:00.601078 12631 log.cc:824] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:00.601838 12632 log.cc:824] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:00.602507 12630 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: No bootstrap required, opened a new log
I20260619 11:10:00.602684 12630 ts_tablet_manager.cc:1397] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:10:00.602886 12631 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: No bootstrap required, opened a new log
I20260619 11:10:00.603052 12631 ts_tablet_manager.cc:1397] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:10:00.603534 12632 tablet_bootstrap.cc:492] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: No bootstrap required, opened a new log
I20260619 11:10:00.603709 12632 ts_tablet_manager.cc:1397] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:10:00.606922 12632 raft_consensus.cc:348] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.607144 12632 raft_consensus.cc:374] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:00.607195 12632 raft_consensus.cc:729] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f647b42ce2f54200a0f7bf632007a6eb, State: Initialized, Role: FOLLOWER
I20260619 11:10:00.607195 12630 raft_consensus.cc:348] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.607195 12631 raft_consensus.cc:348] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.607429 12630 raft_consensus.cc:374] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:00.607429 12631 raft_consensus.cc:374] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:00.607476 12630 raft_consensus.cc:729] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376c1592e62743e2a34add4e24cffd93, State: Initialized, Role: FOLLOWER
I20260619 11:10:00.607486 12631 raft_consensus.cc:729] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5a0cf914a042f4b88b539a189fe434, State: Initialized, Role: FOLLOWER
I20260619 11:10:00.607662 12632 consensus_queue.cc:260] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.607918 12631 consensus_queue.cc:260] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.607918 12630 consensus_queue.cc:260] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.608590 12632 ts_tablet_manager.cc:1428] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:10:00.608806 12630 ts_tablet_manager.cc:1428] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:10:00.608806 12631 ts_tablet_manager.cc:1428] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:10:00.608927 12613 heartbeater.cc:503] Master 127.7.90.126:46717 was elected leader, sending a full tablet report...
I20260619 11:10:00.609105 12482 heartbeater.cc:503] Master 127.7.90.126:46717 was elected leader, sending a full tablet report...
I20260619 11:10:00.609105 12351 heartbeater.cc:503] Master 127.7.90.126:46717 was elected leader, sending a full tablet report...
I20260619 11:10:00.612974 12636 raft_consensus.cc:482] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:00.613114 12636 raft_consensus.cc:504] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.614149 12636 leader_election.cc:288] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d5a0cf914a042f4b88b539a189fe434 (127.7.90.65:44521), 376c1592e62743e2a34add4e24cffd93 (127.7.90.66:42239)
I20260619 11:10:00.615994 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.614596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:55571 (local address 127.7.90.65:44521)
0619 11:10:00.614712 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.614717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.614730 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:00.614764 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.614767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.614796 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.614874 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.615637 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.615641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.615644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.615653 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.615671 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.615679 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.615777 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.615843 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.615880 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:00.615994 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.614346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44521 (local address 127.7.90.67:55571)
0619 11:10:00.614473 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.614512 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.614584 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.614962 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.614966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.614980 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.615493 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.615496 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.615505 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.615545 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.615766 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.615769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.615772 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.615848 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.615880 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:10:00.616192 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.614508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:34135 (local address 127.7.90.66:42239)
0619 11:10:00.614606 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.614612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.614878 (+ 266us) server_negotiation.cc:407] Connection header received
0619 11:10:00.614967 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.614970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.615022 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.615084 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.615832 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.615836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.615839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.615849 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.615866 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.615875 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.615972 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.616045 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.616081 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:10:00.616230 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.614346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:42239 (local address 127.7.90.67:34135)
0619 11:10:00.614806 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.614820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.614894 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.615123 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.615128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.615146 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:00.615684 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:00.615686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:00.615697 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:00.615726 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:00.615962 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:00.615965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:00.615968 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:10:00.616052 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 11:10:00.616081 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":203,"thread_start_us":89,"threads_started":1}
I20260619 11:10:00.616760 12436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaeef5a6cc54d5591528d4cfee3ffe9" candidate_uuid: "f647b42ce2f54200a0f7bf632007a6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376c1592e62743e2a34add4e24cffd93" is_pre_election: true
I20260619 11:10:00.616822 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaeef5a6cc54d5591528d4cfee3ffe9" candidate_uuid: "f647b42ce2f54200a0f7bf632007a6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5a0cf914a042f4b88b539a189fe434" is_pre_election: true
I20260619 11:10:00.617012 12436 raft_consensus.cc:2454] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f647b42ce2f54200a0f7bf632007a6eb in term 0.
I20260619 11:10:00.617015 12305 raft_consensus.cc:2454] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f647b42ce2f54200a0f7bf632007a6eb in term 0.
I20260619 11:10:00.617408 12500 leader_election.cc:302] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376c1592e62743e2a34add4e24cffd93, f647b42ce2f54200a0f7bf632007a6eb; no voters:
I20260619 11:10:00.617663 12636 raft_consensus.cc:2781] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:00.617733 12636 raft_consensus.cc:482] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:00.617763 12636 raft_consensus.cc:3037] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:00.618700 12636 raft_consensus.cc:504] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.619014 12636 leader_election.cc:288] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [CANDIDATE]: Term 1 election: Requested vote from peers 2d5a0cf914a042f4b88b539a189fe434 (127.7.90.65:44521), 376c1592e62743e2a34add4e24cffd93 (127.7.90.66:42239)
I20260619 11:10:00.619324 12436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaeef5a6cc54d5591528d4cfee3ffe9" candidate_uuid: "f647b42ce2f54200a0f7bf632007a6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376c1592e62743e2a34add4e24cffd93"
I20260619 11:10:00.619316 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaeef5a6cc54d5591528d4cfee3ffe9" candidate_uuid: "f647b42ce2f54200a0f7bf632007a6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5a0cf914a042f4b88b539a189fe434"
I20260619 11:10:00.619426 12436 raft_consensus.cc:3037] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:00.619426 12305 raft_consensus.cc:3037] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:00.620561 12436 raft_consensus.cc:2454] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f647b42ce2f54200a0f7bf632007a6eb in term 1.
I20260619 11:10:00.620561 12305 raft_consensus.cc:2454] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f647b42ce2f54200a0f7bf632007a6eb in term 1.
I20260619 11:10:00.620822 12500 leader_election.cc:302] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376c1592e62743e2a34add4e24cffd93, f647b42ce2f54200a0f7bf632007a6eb; no voters:
I20260619 11:10:00.621006 12636 raft_consensus.cc:2781] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:00.621222 12636 raft_consensus.cc:686] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 1 LEADER]: Becoming Leader. State: Replica: f647b42ce2f54200a0f7bf632007a6eb, State: Running, Role: LEADER
I20260619 11:10:00.621493 12636 consensus_queue.cc:237] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } }
I20260619 11:10:00.624207 12163 catalog_manager.cc:5495] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb reported cstate change: term changed from 0 to 1, leader changed from <none> to f647b42ce2f54200a0f7bf632007a6eb (127.7.90.67). New cstate: current_term: 1 leader_uuid: "f647b42ce2f54200a0f7bf632007a6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f647b42ce2f54200a0f7bf632007a6eb" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 43455 } health_report { overall_health: HEALTHY } } }
W20260619 11:10:00.631047 12483 tablet.cc:2369] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:00.638983 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.637668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.7.90.67:43455)
0619 11:10:00.637782 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.637786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.637807 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:10:00.637918 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.637920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.637966 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.638033 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.638643 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.638646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.638648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.638658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.638676 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.638683 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.638750 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.638827 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.638862 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:00.644176 12436 raft_consensus.cc:1261] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 1 FOLLOWER]: Refusing update from remote peer f647b42ce2f54200a0f7bf632007a6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:00.644174 12305 raft_consensus.cc:1261] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 1 FOLLOWER]: Refusing update from remote peer f647b42ce2f54200a0f7bf632007a6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:00.644788 12636 consensus_queue.cc:1035] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5a0cf914a042f4b88b539a189fe434" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:00.644986 12640 consensus_queue.cc:1035] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "376c1592e62743e2a34add4e24cffd93" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:00.651237 12644 mvcc.cc:205] Tried to move back new op lower bound from 7298528873032634368 to 7298528872948027392. Current Snapshot: MvccSnapshot[applied={T|T < 7298528873032634368}]
I20260619 11:10:00.652318 12643 mvcc.cc:205] Tried to move back new op lower bound from 7298528873032634368 to 7298528872948027392. Current Snapshot: MvccSnapshot[applied={T|T < 7298528873032634368}]
I20260619 11:10:00.653820 12645 mvcc.cc:205] Tried to move back new op lower bound from 7298528873032634368 to 7298528872948027392. Current Snapshot: MvccSnapshot[applied={T|T < 7298528873032634368}]
I20260619 11:10:00.661193 12161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:00.661376 12161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:00.663764 12161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d779c210b0a645e398a56b82cf6735b4: Sending DeleteTablet for 3 replicas of tablet aaaeef5a6cc54d5591528d4cfee3ffe9
I20260619 11:10:00.664507 12285 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:00 UTC) from {username='slave'} at 127.0.0.1:39946
I20260619 11:10:00.664561 12416 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:00 UTC) from {username='slave'} at 127.0.0.1:36500
I20260619 11:10:00.664940 12547 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaeef5a6cc54d5591528d4cfee3ffe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:00 UTC) from {username='slave'} at 127.0.0.1:33154
I20260619 11:10:00.665010 12654 tablet_replica.cc:330] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: stopping tablet replica
I20260619 11:10:00.665295 12654 raft_consensus.cc:2229] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:00.665333 12655 tablet_replica.cc:330] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: stopping tablet replica
I20260619 11:10:00.665616 12654 raft_consensus.cc:2258] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:00.665642 12655 raft_consensus.cc:2229] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:00.666082 12653 tablet_replica.cc:330] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434: stopping tablet replica
I20260619 11:10:00.666144 12655 raft_consensus.cc:2258] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:00.666376 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12223
I20260619 11:10:00.666530 12653 raft_consensus.cc:2229] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:00.666827 12653 raft_consensus.cc:2258] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 2d5a0cf914a042f4b88b539a189fe434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:00.668926 12655 ts_tablet_manager.cc:1905] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:00.669972 12654 ts_tablet_manager.cc:1905] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:00.671797 12655 ts_tablet_manager.cc:1918] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:00.671913 12655 log.cc:1196] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/aaaeef5a6cc54d5591528d4cfee3ffe9
I20260619 11:10:00.672276 12655 ts_tablet_manager.cc:1939] T aaaeef5a6cc54d5591528d4cfee3ffe9 P f647b42ce2f54200a0f7bf632007a6eb: Deleting consensus metadata
I20260619 11:10:00.673084 12654 ts_tablet_manager.cc:1918] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:00.673242 12654 log.cc:1196] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/aaaeef5a6cc54d5591528d4cfee3ffe9
I20260619 11:10:00.673537 12148 catalog_manager.cc:4841] TS f647b42ce2f54200a0f7bf632007a6eb (127.7.90.67:43455): tablet aaaeef5a6cc54d5591528d4cfee3ffe9 (table test-table [id=561778e048324ddba5a87e442d6d83c6]) successfully deleted
I20260619 11:10:00.673727 12654 ts_tablet_manager.cc:1939] T aaaeef5a6cc54d5591528d4cfee3ffe9 P 376c1592e62743e2a34add4e24cffd93: Deleting consensus metadata
W20260619 11:10:00.674638 12149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:44521 (error 108)
I20260619 11:10:00.674736 12147 catalog_manager.cc:4841] TS 376c1592e62743e2a34add4e24cffd93 (127.7.90.66:42239): tablet aaaeef5a6cc54d5591528d4cfee3ffe9 (table test-table [id=561778e048324ddba5a87e442d6d83c6]) successfully deleted
I20260619 11:10:00.675400 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12354
I20260619 11:10:00.675626 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.675351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44521 (local address 127.0.0.1:39954)
0619 11:10:00.675495 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.675556 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:44521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 11:10:00.682049 12149 catalog_manager.cc:4584] TS 2d5a0cf914a042f4b88b539a189fe434 (127.7.90.65:44521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaaeef5a6cc54d5591528d4cfee3ffe9: Network error: Client connection negotiation failed: client connection to 127.7.90.65:44521: connect: Connection refused (error 111)
I20260619 11:10:00.682754 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12485
I20260619 11:10:00.687646 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12132
2026-06-19T11:10:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (902 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 11:10:00.708714 7529 mini_oidc.cc:152] Starting JWKS server
I20260619 11:10:00.710402 7529 webserver.cc:460] Webserver started at https://127.0.0.1:42157/ using document root <none> and password file <none>
I20260619 11:10:00.710533 7529 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:10:00.710750 7529 webserver.cc:460] Webserver started at http://127.0.0.1:45051/ using document root <none> and password file <none>
2026-06-19T11:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:00Z Disabled control of system clock
I20260619 11:10:00.719641 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43661
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:43661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/testchainca.cert with env {}
W20260619 11:10:00.825652 12662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:00.825883 12662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:00.825927 12662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:00.829725 12662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:00.829768 12662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:00.829785 12662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:00.829802 12662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:00.829816 12662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:10:00.829831 12662 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:10:00.829845 12662 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42157/jwks.json
I20260619 11:10:00.834815 12662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:43661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:00.835702 12662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:00.836592 12662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:00.844480 12670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.844480 12668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:00.844501 12667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:00.844579 12662 server_base.cc:1034] running on GCE node
I20260619 11:10:00.845285 12662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:00.846371 12662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:00.847563 12662 hybrid_clock.cc:648] HybridClock initialized: now 1781867400847538 us; error 45 us; skew 500 ppm
I20260619 11:10:00.849299 12662 webserver.cc:460] Webserver started at http://127.7.90.126:33979/ using document root <none> and password file <none>
I20260619 11:10:00.849900 12662 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:00.849996 12662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:00.850206 12662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:00.852138 12662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "621cfbfb294546f99c9022c6f1681602"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "87a76c029e9927c56430fa9e0ac5811c"
server_key_iv: "cf49052b95182dca02ba688456e5262c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.852685 12662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "621cfbfb294546f99c9022c6f1681602"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "87a76c029e9927c56430fa9e0ac5811c"
server_key_iv: "cf49052b95182dca02ba688456e5262c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.856182 12662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:10:00.858650 12676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:00.859854 12662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:00.860031 12662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "621cfbfb294546f99c9022c6f1681602"
format_stamp: "Formatted at 2026-06-19 11:10:00 on dist-test-slave-f1t5"
server_key: "87a76c029e9927c56430fa9e0ac5811c"
server_key_iv: "cf49052b95182dca02ba688456e5262c"
server_key_version: "encryptionkey@0"
I20260619 11:10:00.860148 12662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:00.881105 12662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:00.881906 12662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:00.882124 12662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:00.889760 12728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:43661 every 8 connection(s)
I20260619 11:10:00.889756 12662 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:43661
I20260619 11:10:00.890801 12662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:00.894289 12729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:00.895176 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12662
I20260619 11:10:00.895334 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:00.895640 7529 external_mini_cluster.cc:1428] Setting key ad8d4628b4b30def4e1ad0b420efab36
I20260619 11:10:00.900360 12729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602: Bootstrap starting.
I20260619 11:10:00.902652 12729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:00.903401 12729 log.cc:824] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:00.905475 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.897116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.7.90.126:43661)
0619 11:10:00.897843 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.897861 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.897924 (+ 63us) server_negotiation.cc:407] Connection header received
0619 11:10:00.898593 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.898610 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.899056 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.899405 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:00.899841 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.900881 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.901874 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.902371 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.902564 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:00.903476 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:00.903482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:00.903495 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:00.904036 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:00.904113 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:00.904145 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:00.904254 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:00.904436 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 11:10:00.904733 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260619 11:10:00.905475 12729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602: No bootstrap required, opened a new log
I20260619 11:10:00.908365 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:10:00.909443 12729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } }
I20260619 11:10:00.909741 12729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:00.909818 12729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621cfbfb294546f99c9022c6f1681602, State: Initialized, Role: FOLLOWER
I20260619 11:10:00.910218 12729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } }
I20260619 11:10:00.910356 12729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:00.910418 12729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:00.910566 12729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:00.911485 12729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } }
I20260619 11:10:00.911829 12729 leader_election.cc:302] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 621cfbfb294546f99c9022c6f1681602; no voters:
I20260619 11:10:00.912127 12729 leader_election.cc:288] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:00.912271 12734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:00.912505 12734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [term 1 LEADER]: Becoming Leader. State: Replica: 621cfbfb294546f99c9022c6f1681602, State: Running, Role: LEADER
I20260619 11:10:00.912844 12734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } }
I20260619 11:10:00.913307 12729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:00.914443 12735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "621cfbfb294546f99c9022c6f1681602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } } }
I20260619 11:10:00.914605 12735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:00.914885 12736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 621cfbfb294546f99c9022c6f1681602. Latest consensus state: current_term: 1 leader_uuid: "621cfbfb294546f99c9022c6f1681602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "621cfbfb294546f99c9022c6f1681602" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43661 } } }
I20260619 11:10:00.914973 12736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:00.915649 12743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:00.918524 12743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:00.923632 12743 catalog_manager.cc:1347] Generated new cluster ID: e41dc95c72e940bab515a56a625d9bad
I20260619 11:10:00.923722 12743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:00.938681 12743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:00.939898 12743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:00.957762 12743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 621cfbfb294546f99c9022c6f1681602: Generated new TSK 0
I20260619 11:10:00.958649 12743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:00.999796 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:00.979861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.7.90.126:43661)
0619 11:10:00.979998 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:00.980002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:00.980164 (+ 162us) server_negotiation.cc:407] Connection header received
0619 11:10:00.980340 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:00.980343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:00.980398 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:00.980534 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:10:00.980742 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.981590 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.982468 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.982825 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.982928 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:00.983216 (+ 288us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:10:00.999427 (+ 16211us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:10:00.999671 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 11:10:00.999902 12732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
E20260619 11:10:01.000247 12759 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 11:10:01.000371 12759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 11:10:01.001212 12758 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:00.979764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43661 (local address 127.0.0.1:53206)
0619 11:10:00.980071 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:00.980085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:00.980188 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:00.980531 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:00.980533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:00.980540 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:10:00.980638 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:00.980642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.981701 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.981704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:00.982357 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:00.982364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:00.982935 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:00.982938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:00.983002 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:00.983042 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:10:00.999659 (+ 16617us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:10:00.999905 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.90.126:43661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.90.126:43661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 11:10:01.001955 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12662
2026-06-19T11:10:01Z chronyd exiting
I20260619 11:10:01.038558 7529 test_util.cc:175] -----------------------------------------------
I20260619 11:10:01.038637 7529 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867375844525-7529-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 11:10:01.039616 7529 mini_oidc.cc:152] Starting JWKS server
I20260619 11:10:01.040395 7529 webserver.cc:460] Webserver started at https://127.0.0.1:43699/ using document root <none> and password file <none>
I20260619 11:10:01.040491 7529 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:10:01.040767 7529 webserver.cc:460] Webserver started at http://127.0.0.1:42985/ using document root <none> and password file <none>
2026-06-19T11:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:01Z Disabled control of system clock
I20260619 11:10:01.049737 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37905
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43699/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/testchainca.cert with env {}
W20260619 11:10:01.158955 12765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:01.159225 12765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:01.159265 12765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:01.163158 12765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:01.163209 12765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:01.163224 12765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:01.163241 12765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:01.163256 12765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:10:01.163272 12765 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:10:01.163285 12765 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43699/jwks.json
I20260619 11:10:01.168455 12765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43699/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:01.169420 12765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:01.170373 12765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:10:01.178728 12765 server_base.cc:1034] running on GCE node
W20260619 11:10:01.178696 12773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:01.178747 12770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:01.178889 12771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:01.179454 12765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:01.180485 12765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:01.181665 12765 hybrid_clock.cc:648] HybridClock initialized: now 1781867401181655 us; error 29 us; skew 500 ppm
I20260619 11:10:01.183471 12765 webserver.cc:460] Webserver started at http://127.7.90.126:41587/ using document root <none> and password file <none>
I20260619 11:10:01.184034 12765 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:01.184134 12765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:01.184355 12765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:01.186389 12765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "3b712667380a4784be1e06051a54016c"
server_key_iv: "7b1b19ac38e8f16c8ac0679e0cbcea58"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.186935 12765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "3b712667380a4784be1e06051a54016c"
server_key_iv: "7b1b19ac38e8f16c8ac0679e0cbcea58"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.190620 12765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:10:01.192875 12779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:01.194100 12765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:10:01.194234 12765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "3b712667380a4784be1e06051a54016c"
server_key_iv: "7b1b19ac38e8f16c8ac0679e0cbcea58"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.194342 12765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:01.217725 12765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:01.218451 12765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:01.218657 12765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:01.225281 12765 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37905
I20260619 11:10:01.225366 12831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37905 every 8 connection(s)
I20260619 11:10:01.226260 12765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:01.229241 12832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:01.234630 12832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3: Bootstrap starting.
I20260619 11:10:01.235666 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12765
I20260619 11:10:01.235791 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:01.236109 7529 external_mini_cluster.cc:1428] Setting key 115b0c4d12206dae94342c2f307e2b46
I20260619 11:10:01.237565 12832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:01.238309 12832 log.cc:824] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:01.240154 12832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3: No bootstrap required, opened a new log
I20260619 11:10:01.243572 12832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } }
I20260619 11:10:01.243779 12832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:01.243847 12832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96dd3c70827b49b087fdf1c3c6b9fbe3, State: Initialized, Role: FOLLOWER
I20260619 11:10:01.244349 12832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } }
I20260619 11:10:01.244745 12832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:01.244876 12832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:01.245028 12832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:01.245621 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.237221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.7.90.126:37905)
0619 11:10:01.237941 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.237957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.237991 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:10:01.238673 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.238691 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.239022 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.239352 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:01.239774 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.240869 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.241873 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.242350 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.242534 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.243378 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:01.243383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:01.243396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:01.243815 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:01.243886 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:01.243931 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:01.244043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:01.244180 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 11:10:01.244433 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":136,"threads_started":1}
I20260619 11:10:01.246131 12832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } }
I20260619 11:10:01.246487 12832 leader_election.cc:302] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96dd3c70827b49b087fdf1c3c6b9fbe3; no voters:
I20260619 11:10:01.246840 12832 leader_election.cc:288] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:01.247057 12837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:01.247277 12837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 96dd3c70827b49b087fdf1c3c6b9fbe3, State: Running, Role: LEADER
I20260619 11:10:01.247575 12837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } }
I20260619 11:10:01.248174 12832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:01.248557 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:10:01.248737 12837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96dd3c70827b49b087fdf1c3c6b9fbe3. Latest consensus state: current_term: 1 leader_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } } }
I20260619 11:10:01.248845 12837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.249125 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96dd3c70827b49b087fdf1c3c6b9fbe3" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37905 } } }
I20260619 11:10:01.249205 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.249864 12842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:01.253125 12842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:01.258541 12842 catalog_manager.cc:1347] Generated new cluster ID: dbc646b989854db69e8e3d9492c62381
I20260619 11:10:01.258637 12842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:01.287750 12842 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:01.288925 12842 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:01.296844 12842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96dd3c70827b49b087fdf1c3c6b9fbe3: Generated new TSK 0
I20260619 11:10:01.297640 12842 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 11:10:01.323694 12862 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 11:10:01.323792 12862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 11:10:01.324470 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.308728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.7.90.126:37905)
0619 11:10:01.308860 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.308865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.309031 (+ 166us) server_negotiation.cc:407] Connection header received
0619 11:10:01.309156 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.309159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.309213 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.309326 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:10:01.309549 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.310363 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.311205 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.311562 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.311662 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.311934 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:10:01.324076 (+ 12142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:10:01.324331 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:10:01.324545 12835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 11:10:01.324565 12861 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:01.308617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37905 (local address 127.0.0.1:54160)
0619 11:10:01.308929 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:01.308940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:01.309052 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:01.309327 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:01.309329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:01.309335 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:10:01.309427 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.309430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.310467 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.310470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.311092 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.311099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.311679 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.311683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.311747 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.311769 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:10:01.324256 (+ 12487us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:10:01.324403 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.90.126:37905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260619 11:10:01.325351 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12765
2026-06-19T11:10:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 11:10:01.364391 7529 mini_oidc.cc:152] Starting JWKS server
I20260619 11:10:01.365242 7529 webserver.cc:460] Webserver started at https://127.0.0.1:37221/ using document root <none> and password file <none>
I20260619 11:10:01.365368 7529 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:10:01.365679 7529 webserver.cc:460] Webserver started at http://127.0.0.1:40597/ using document root <none> and password file <none>
2026-06-19T11:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:01Z Disabled control of system clock
I20260619 11:10:01.374907 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:34619
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:34619
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37221/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:10:01.482942 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:01.483198 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:01.483237 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:01.487211 12868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:01.487267 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:01.487282 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:01.487298 12868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:01.487344 12868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:10:01.487377 12868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:10:01.487407 12868 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37221/jwks.json
I20260619 11:10:01.492323 12868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:34619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37221/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:01.493427 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:01.494851 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:01.503330 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:01.503330 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:01.503398 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:01.503665 12868 server_base.cc:1034] running on GCE node
I20260619 11:10:01.504238 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:01.505259 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:01.506453 12868 hybrid_clock.cc:648] HybridClock initialized: now 1781867401506417 us; error 49 us; skew 500 ppm
I20260619 11:10:01.508292 12868 webserver.cc:460] Webserver started at http://127.7.90.126:43551/ using document root <none> and password file <none>
I20260619 11:10:01.508888 12868 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:01.508951 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:01.509207 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:01.511420 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "b1b92e16d5ba41818f892a8e9877818e"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "2888b29db26a355eea60695a2bf3c464"
server_key_iv: "6a4e6e9defefce2261e2628a1d40eec8"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.512023 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "b1b92e16d5ba41818f892a8e9877818e"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "2888b29db26a355eea60695a2bf3c464"
server_key_iv: "6a4e6e9defefce2261e2628a1d40eec8"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.515853 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:10:01.518373 12882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:01.519577 12868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:01.519721 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "b1b92e16d5ba41818f892a8e9877818e"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "2888b29db26a355eea60695a2bf3c464"
server_key_iv: "6a4e6e9defefce2261e2628a1d40eec8"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.519847 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:01.529770 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:01.530409 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:01.530546 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:01.537225 12868 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:34619
I20260619 11:10:01.537274 12934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:34619 every 8 connection(s)
I20260619 11:10:01.538496 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:01.540380 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12868
I20260619 11:10:01.540510 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:01.540812 7529 external_mini_cluster.cc:1428] Setting key 02a298b798401f74c04a437001d9ee4e
I20260619 11:10:01.542491 12935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:01.548628 12935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e: Bootstrap starting.
I20260619 11:10:01.549677 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.542080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.7.90.126:34619)
0619 11:10:01.542685 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.542700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.542733 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:10:01.543385 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.543402 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.543780 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.544064 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:01.544389 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.545342 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.546505 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.546995 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.547124 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.547988 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:01.547993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:01.548005 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:01.548365 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:01.548416 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:01.548434 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:01.548506 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:01.548657 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 11:10:01.548917 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260619 11:10:01.551017 12935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:01.551857 12935 log.cc:824] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:01.552525 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:10:01.553987 12935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e: No bootstrap required, opened a new log
I20260619 11:10:01.557152 12935 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } }
I20260619 11:10:01.557430 12935 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:01.557490 12935 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b92e16d5ba41818f892a8e9877818e, State: Initialized, Role: FOLLOWER
I20260619 11:10:01.557953 12935 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } }
I20260619 11:10:01.558109 12935 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:01.558204 12935 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:01.558300 12935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:01.559317 12935 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } }
I20260619 11:10:01.559645 12935 leader_election.cc:302] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b92e16d5ba41818f892a8e9877818e; no voters:
I20260619 11:10:01.559937 12935 leader_election.cc:288] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:01.560092 12940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:01.560345 12940 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [term 1 LEADER]: Becoming Leader. State: Replica: b1b92e16d5ba41818f892a8e9877818e, State: Running, Role: LEADER
I20260619 11:10:01.560633 12940 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } }
I20260619 11:10:01.561093 12935 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:01.562177 12942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b92e16d5ba41818f892a8e9877818e. Latest consensus state: current_term: 1 leader_uuid: "b1b92e16d5ba41818f892a8e9877818e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } } }
I20260619 11:10:01.562206 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b92e16d5ba41818f892a8e9877818e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b92e16d5ba41818f892a8e9877818e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34619 } } }
I20260619 11:10:01.562310 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.562310 12942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.562783 12949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:01.565338 12949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:01.570621 12949 catalog_manager.cc:1347] Generated new cluster ID: 323e74daa25c49f4a6278a46c7ca1f21
I20260619 11:10:01.570694 12949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:01.589196 12949 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:01.590162 12949 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:01.597481 12949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b92e16d5ba41818f892a8e9877818e: Generated new TSK 0
I20260619 11:10:01.598160 12949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:01.615124 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.610101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.7.90.126:34619)
0619 11:10:01.610241 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.610248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.610410 (+ 162us) server_negotiation.cc:407] Connection header received
0619 11:10:01.610581 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.610584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.610628 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.610711 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:10:01.610975 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.611490 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.612721 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.612923 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.612995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.613324 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:10:01.614765 (+ 1441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37221/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:10:01.614988 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37221/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 11:10:01.615195 12938 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37221/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 11:10:01.616992 12964 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:01.610018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:34619 (local address 127.0.0.1:52560)
0619 11:10:01.610314 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:01.610330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:01.610448 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:01.610745 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:01.610748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:01.610756 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:10:01.610874 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.610879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.611606 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.611610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.612605 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.612615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.613041 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.613044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.613133 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.613160 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:10:01.614937 (+ 1777us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37221/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:10:01.616809 (+ 1872us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.90.126:34619: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37221/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1668,"thread_start_us":108,"threads_started":1}
I20260619 11:10:01.617695 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12868
E20260619 11:10:01.627996 12965 webserver.cc:533] Webserver: sslize failed:
E20260619 11:10:01.628068 12965 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T11:10:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T11:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:01Z Disabled control of system clock
I20260619 11:10:01.665598 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36301
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:36301
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/testchainca.cert with env {}
W20260619 11:10:01.776628 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:01.776892 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:01.776933 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:01.781037 12969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:01.781100 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:01.781116 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:01.781132 12969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:01.781148 12969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:10:01.781163 12969 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:10:01.781177 12969 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 11:10:01.787108 12969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:36301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:01.788149 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:01.789129 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:01.797710 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:01.797724 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:01.797775 12969 server_base.cc:1034] running on GCE node
W20260619 11:10:01.797731 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:01.798478 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:01.799598 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:01.800781 12969 hybrid_clock.cc:648] HybridClock initialized: now 1781867401800765 us; error 27 us; skew 500 ppm
I20260619 11:10:01.802650 12969 webserver.cc:460] Webserver started at http://127.7.90.126:42099/ using document root <none> and password file <none>
I20260619 11:10:01.803229 12969 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:01.803323 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:01.803551 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:01.805609 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "19d40836fd194163ae576a93efda29fb"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "8eb6d2eacaa7f500f9e5eb28f1fec219"
server_key_iv: "e7790493a0c450aa864de35a4e4fc0bd"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.806188 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "19d40836fd194163ae576a93efda29fb"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "8eb6d2eacaa7f500f9e5eb28f1fec219"
server_key_iv: "e7790493a0c450aa864de35a4e4fc0bd"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.809863 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:01.812394 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:01.813760 12969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:01.813899 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "19d40836fd194163ae576a93efda29fb"
format_stamp: "Formatted at 2026-06-19 11:10:01 on dist-test-slave-f1t5"
server_key: "8eb6d2eacaa7f500f9e5eb28f1fec219"
server_key_iv: "e7790493a0c450aa864de35a4e4fc0bd"
server_key_version: "encryptionkey@0"
I20260619 11:10:01.814020 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:01.843717 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:01.844453 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:01.844661 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:01.853456 12969 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:36301
I20260619 11:10:01.853485 13035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:36301 every 8 connection(s)
I20260619 11:10:01.854663 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:01.857613 13036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:01.861410 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 12969
I20260619 11:10:01.861562 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:01.861855 7529 external_mini_cluster.cc:1428] Setting key a49cf8c0e08ddf2ad3cfc102dbd4e833
I20260619 11:10:01.863417 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb: Bootstrap starting.
I20260619 11:10:01.865760 13036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:01.866475 13036 log.cc:824] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:01.868522 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb: No bootstrap required, opened a new log
I20260619 11:10:01.871212 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.863228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.7.90.126:36301)
0619 11:10:01.863845 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.863861 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.863896 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:10:01.864512 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.864528 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.864949 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.865230 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:01.865641 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.866705 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.867801 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.868255 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.868378 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.869211 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:01.869216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:01.869228 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:01.869677 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:01.869732 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:01.869751 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:01.869826 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:01.869982 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:10:01.870262 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260619 11:10:01.871764 13036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } }
I20260619 11:10:01.872008 13036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:01.872083 13036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d40836fd194163ae576a93efda29fb, State: Initialized, Role: FOLLOWER
I20260619 11:10:01.872505 13036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } }
I20260619 11:10:01.872648 13036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:01.872732 13036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:01.872846 13036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:01.873909 13036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } }
I20260619 11:10:01.874114 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:10:01.874312 13036 leader_election.cc:302] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d40836fd194163ae576a93efda29fb; no voters:
I20260619 11:10:01.874626 13036 leader_election.cc:288] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:01.874748 13041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:01.874972 13041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [term 1 LEADER]: Becoming Leader. State: Replica: 19d40836fd194163ae576a93efda29fb, State: Running, Role: LEADER
I20260619 11:10:01.875303 13041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } }
I20260619 11:10:01.875886 13036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:01.877032 13043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d40836fd194163ae576a93efda29fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } } }
I20260619 11:10:01.877039 13042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d40836fd194163ae576a93efda29fb. Latest consensus state: current_term: 1 leader_uuid: "19d40836fd194163ae576a93efda29fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d40836fd194163ae576a93efda29fb" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36301 } } }
I20260619 11:10:01.877246 13042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.877276 13043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:01.878113 13050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:01.880614 13050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:01.885799 13050 catalog_manager.cc:1347] Generated new cluster ID: 38138df6fcc445679f7fff8224b96853
I20260619 11:10:01.885869 13050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:01.925648 13050 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:01.926586 13050 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:01.932984 13050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d40836fd194163ae576a93efda29fb: Generated new TSK 0
I20260619 11:10:01.933743 13050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:02.379374 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:01.946591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.7.90.126:36301)
0619 11:10:01.946731 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:01.946736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:01.946888 (+ 152us) server_negotiation.cc:407] Connection header received
0619 11:10:01.947039 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:01.947041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:01.947086 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:01.947174 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:10:01.947422 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.947929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.949153 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.949360 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.949432 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.949789 (+ 357us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:10:02.378999 (+429210us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:10:02.379255 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 11:10:02.379446 13039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51752: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 11:10:02.379669 13065 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:01.946511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:36301 (local address 127.0.0.1:51752)
0619 11:10:01.946792 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:01.946806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:01.946917 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:01.947201 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:01.947205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:01.947212 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:10:01.947320 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.947326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.948045 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.948049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.949038 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:01.949048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:01.949474 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:01.949478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:01.949598 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:01.949625 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:10:02.379228 (+429603us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:10:02.379412 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.90.126:36301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20260619 11:10:02.380286 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 12969
2026-06-19T11:10:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (741 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (143 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:02 dist-test-slave-f1t5 krb5kdc[13077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:02 dist-test-slave-f1t5 krb5kdc[13077](info): set up 2 sockets
Jun 19 11:10:02 dist-test-slave-f1t5 krb5kdc[13077](info): commencing operation
krb5kdc: starting...
W20260619 11:10:03.681988 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:03 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:10:03.813974 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40611
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:03.934067 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:03.934388 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:03.934432 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:03.938761 13090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:03.938848 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:03.938868 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:03.938885 13090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:03.938901 13090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:03.944696 13090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:03.946030 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:03.947176 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:03.956748 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:03.956748 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:03.956751 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:03.956981 13090 server_base.cc:1034] running on GCE node
I20260619 11:10:03.957805 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:03.959021 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:03.960218 13090 hybrid_clock.cc:648] HybridClock initialized: now 1781867403960207 us; error 28 us; skew 500 ppm
Jun 19 11:10:03 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:03.966349 13090 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:10:03.967442 13090 webserver.cc:460] Webserver started at http://127.7.90.126:38867/ using document root <none> and password file <none>
I20260619 11:10:03.968075 13090 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:03.968144 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:03.968381 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:03.970703 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "4de154c6fe4a41969cc358a443500339"
format_stamp: "Formatted at 2026-06-19 11:10:03 on dist-test-slave-f1t5"
server_key: "d71a1ac214901221ca2ef6894cdd8add"
server_key_iv: "50001ed693ac1c60da0e0ad7703a450e"
server_key_version: "encryptionkey@0"
I20260619 11:10:03.971323 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "4de154c6fe4a41969cc358a443500339"
format_stamp: "Formatted at 2026-06-19 11:10:03 on dist-test-slave-f1t5"
server_key: "d71a1ac214901221ca2ef6894cdd8add"
server_key_iv: "50001ed693ac1c60da0e0ad7703a450e"
server_key_version: "encryptionkey@0"
I20260619 11:10:03.975586 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:10:03.978528 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:03.980053 13090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:03.980180 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "4de154c6fe4a41969cc358a443500339"
format_stamp: "Formatted at 2026-06-19 11:10:03 on dist-test-slave-f1t5"
server_key: "d71a1ac214901221ca2ef6894cdd8add"
server_key_iv: "50001ed693ac1c60da0e0ad7703a450e"
server_key_version: "encryptionkey@0"
I20260619 11:10:03.980351 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:04.003957 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:04.008101 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:04.008391 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:04.017038 13090 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40611
I20260619 11:10:04.017050 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40611 every 8 connection(s)
I20260619 11:10:04.018316 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:04.021405 13158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:04.021409 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13090
I20260619 11:10:04.021618 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:04.021927 7529 external_mini_cluster.cc:1428] Setting key fd3030e83eba380be004dca366f7a0f7
I20260619 11:10:04.028283 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339: Bootstrap starting.
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:04.031245 13158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:04.032007 13158 log.cc:824] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:04.034305 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339: No bootstrap required, opened a new log
I20260619 11:10:04.036727 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.023343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.7.90.126:40611)
0619 11:10:04.023766 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.023786 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.023832 (+ 46us) server_negotiation.cc:407] Connection header received
0619 11:10:04.024568 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.024587 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.024943 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.025234 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.026201 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.027203 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.028347 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.028895 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.029032 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.031887 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.031892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.031907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.031936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.033856 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.034328 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.034339 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.034344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.034513 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.034807 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.034810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.034812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.035213 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.035418 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.035715 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.036007 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":151,"threads_started":1}
I20260619 11:10:04.038175 13158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } }
I20260619 11:10:04.038441 13158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:04.038487 13158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de154c6fe4a41969cc358a443500339, State: Initialized, Role: FOLLOWER
I20260619 11:10:04.038969 13158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } }
I20260619 11:10:04.039130 13158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:04.039197 13158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:04.039274 13158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:04.040132 13158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } }
I20260619 11:10:04.040416 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de154c6fe4a41969cc358a443500339; no voters:
I20260619 11:10:04.040679 13158 leader_election.cc:288] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:04.040802 13163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:04.041064 13163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [term 1 LEADER]: Becoming Leader. State: Replica: 4de154c6fe4a41969cc358a443500339, State: Running, Role: LEADER
I20260619 11:10:04.041430 13163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } }
I20260619 11:10:04.041957 13158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:04.043238 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de154c6fe4a41969cc358a443500339. Latest consensus state: current_term: 1 leader_uuid: "4de154c6fe4a41969cc358a443500339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } } }
I20260619 11:10:04.043444 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:04.043249 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de154c6fe4a41969cc358a443500339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de154c6fe4a41969cc358a443500339" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40611 } } }
I20260619 11:10:04.044000 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:04.044473 13174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:04.047145 13174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:04.052805 13174 catalog_manager.cc:1347] Generated new cluster ID: 93517ee48db34e948c78526d3ddf54a7
I20260619 11:10:04.052886 13174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:04.075295 13174 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:04.076639 13174 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:04.088553 13174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de154c6fe4a41969cc358a443500339: Generated new TSK 0
I20260619 11:10:04.089435 13174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:10:04.120723 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:04.230764 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:04.231019 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:04.231060 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:04.235023 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:04.235075 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:04.235131 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:04.240198 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:04.241154 13184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:04.242178 13184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:04.251880 13189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:04.251909 13190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:04.251909 13192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:04.252036 13184 server_base.cc:1034] running on GCE node
I20260619 11:10:04.252501 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:04.253080 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:04.254236 13184 hybrid_clock.cc:648] HybridClock initialized: now 1781867404254195 us; error 52 us; skew 500 ppm
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.256971 13184 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:10:04.257931 13184 webserver.cc:460] Webserver started at http://127.7.90.65:46233/ using document root <none> and password file <none>
I20260619 11:10:04.258445 13184 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:04.258514 13184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:04.258682 13184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:04.260659 13184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "df4fad16625845539a011de3981691d5"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "5d9de4db16fc4869a6c7cd175b3d8aaa"
server_key_iv: "5a74098ae3518675af076e70006b04bb"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.261165 13184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "df4fad16625845539a011de3981691d5"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "5d9de4db16fc4869a6c7cd175b3d8aaa"
server_key_iv: "5a74098ae3518675af076e70006b04bb"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.264835 13184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:10:04.267302 13199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.268617 13184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:04.268718 13184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "df4fad16625845539a011de3981691d5"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "5d9de4db16fc4869a6c7cd175b3d8aaa"
server_key_iv: "5a74098ae3518675af076e70006b04bb"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.268805 13184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:04.285669 13184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:04.288658 13184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:04.288806 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:04.289310 13184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:04.290088 13184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:04.290134 13184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.290174 13184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:04.290198 13184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.299618 13184 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:33761
I20260619 11:10:04.299656 13312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:33761 every 8 connection(s)
I20260619 11:10:04.300583 13184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:04.307053 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13184
I20260619 11:10:04.307212 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:04.307497 7529 external_mini_cluster.cc:1428] Setting key 77b7cef13cd662438cede73d7117a080
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:04.315505 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.302619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:36635 (local address 127.7.90.126:40611)
0619 11:10:04.302787 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.302791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.303686 (+ 895us) server_negotiation.cc:407] Connection header received
0619 11:10:04.305097 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.305101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.305161 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.305268 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.306476 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.307004 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.308412 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.308656 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.308775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.311305 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.311312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.311318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.311352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.313012 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.313765 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.313771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.313776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.313876 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.314254 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.314261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.314265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.314532 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.314657 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.315177 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.315303 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 11:10:04.316206 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.303036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40611 (local address 127.7.90.65:36635)
0619 11:10:04.303509 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:04.303553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:04.304863 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:04.305443 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:04.305450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:04.305849 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:04.306321 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.306333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.307155 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.307163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.308286 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.308295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.308815 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.308819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.308990 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.309641 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:04.309658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:04.311134 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:04.313222 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.313227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.313239 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.313488 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.314005 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.314008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.314009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.314141 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.314647 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:04.314652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:04.314900 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 11:10:04.315060 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 11:10:04.315304 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:10:04.317600 13313 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40611
I20260619 11:10:04.317816 13313 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:04.318327 13313 heartbeater.cc:511] Master 127.7.90.126:40611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:10:04.319805 13122 ts_manager.cc:194] Registered new tserver with Master: df4fad16625845539a011de3981691d5 (127.7.90.65:33761)
I20260619 11:10:04.321601 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:36635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:10:04.336227 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:04.446630 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:04.446885 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:04.446925 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:04.451037 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:04.451113 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:04.451172 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:04.456419 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:04.457561 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:04.458691 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:04.468838 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:04.468838 13326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:04.468885 13318 server_base.cc:1034] running on GCE node
W20260619 11:10:04.468904 13324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:04.469498 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:04.470198 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:04.471390 13318 hybrid_clock.cc:648] HybridClock initialized: now 1781867404471371 us; error 27 us; skew 500 ppm
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.474287 13318 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:10:04.475164 13318 webserver.cc:460] Webserver started at http://127.7.90.66:36193/ using document root <none> and password file <none>
I20260619 11:10:04.475744 13318 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:04.475836 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:04.476071 13318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:04.478212 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "921e5ec5176149e99bcfe38f0479772e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "1fbbf8163e36663e409bab9c28eaf556"
server_key_iv: "5257a4b7a082251e4817af577106b0f8"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.478781 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "921e5ec5176149e99bcfe38f0479772e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "1fbbf8163e36663e409bab9c28eaf556"
server_key_iv: "5257a4b7a082251e4817af577106b0f8"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.482494 13318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:04.484938 13333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.486272 13318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:10:04.486402 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "921e5ec5176149e99bcfe38f0479772e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "1fbbf8163e36663e409bab9c28eaf556"
server_key_iv: "5257a4b7a082251e4817af577106b0f8"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.486522 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:04.507543 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:04.510646 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:04.510866 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:04.511488 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:04.512333 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:04.512408 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.512483 13318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:04.512527 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.521265 13318 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35851
I20260619 11:10:04.521287 13446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35851 every 8 connection(s)
I20260619 11:10:04.522213 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:04.522820 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13318
I20260619 11:10:04.522923 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:04.523241 7529 external_mini_cluster.cc:1428] Setting key 3591d23c141c4c146ab181b602c0df7c
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:04.541819 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.524205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:58585 (local address 127.7.90.126:40611)
0619 11:10:04.524356 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.524361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.525358 (+ 997us) server_negotiation.cc:407] Connection header received
0619 11:10:04.527148 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.527151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.527195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.527259 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.528905 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.529447 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.531392 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.531728 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.531815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.537390 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.537395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.537398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.537432 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.539323 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.539901 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.539908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.539913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.540032 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.540399 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.540405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.540410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.540712 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.540837 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.541361 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.541539 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:10:04.542642 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.524686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40611 (local address 127.7.90.66:58585)
0619 11:10:04.525181 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:04.525235 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:04.526881 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:04.527465 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:04.527477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:04.528017 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:04.528734 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.528757 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.529633 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.529638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.531212 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.531228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.531905 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.531910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.532121 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.532732 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:04.532758 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:04.537175 (+ 4417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:04.539462 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.539467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.539482 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.539761 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.540132 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.540134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.540136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.540271 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.540820 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:04.540826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:04.541081 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 11:10:04.541238 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 11:10:04.541622 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260619 11:10:04.544116 13447 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40611
I20260619 11:10:04.544425 13447 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:04.545104 13447 heartbeater.cc:511] Master 127.7.90.126:40611 requested a full tablet report, sending...
I20260619 11:10:04.546325 13122 ts_manager.cc:194] Registered new tserver with Master: 921e5ec5176149e99bcfe38f0479772e (127.7.90.66:35851)
I20260619 11:10:04.546931 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:58585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:10:04.556226 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:04.665845 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:04.666121 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:04.666167 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:04.670099 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:04.670159 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:04.670220 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:04.675547 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:04.676568 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:04.677595 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:04.687398 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:04.687536 13452 server_base.cc:1034] running on GCE node
W20260619 11:10:04.687433 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:04.687739 13460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:04.688187 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:04.688872 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:04.690037 13452 hybrid_clock.cc:648] HybridClock initialized: now 1781867404690024 us; error 73 us; skew 500 ppm
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.692832 13452 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:10:04.693887 13452 webserver.cc:460] Webserver started at http://127.7.90.67:33129/ using document root <none> and password file <none>
I20260619 11:10:04.694474 13452 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:04.694569 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:04.694792 13452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:04.696928 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "05fbfdca285b41cc8a1e01cf402bf97e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "0071d92b92ed1069d96c7cccb8265610"
server_key_iv: "fe96dfbe2372f6d662e2c9c3ff60ba71"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.697649 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "05fbfdca285b41cc8a1e01cf402bf97e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "0071d92b92ed1069d96c7cccb8265610"
server_key_iv: "fe96dfbe2372f6d662e2c9c3ff60ba71"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.701359 13452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:04.703912 13467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.705235 13452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:04.705381 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "05fbfdca285b41cc8a1e01cf402bf97e"
format_stamp: "Formatted at 2026-06-19 11:10:04 on dist-test-slave-f1t5"
server_key: "0071d92b92ed1069d96c7cccb8265610"
server_key_iv: "fe96dfbe2372f6d662e2c9c3ff60ba71"
server_key_version: "encryptionkey@0"
I20260619 11:10:04.705502 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:04.728240 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:04.731451 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:04.731675 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:04.732281 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:04.733120 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:04.733197 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.733265 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:04.733317 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:04.742367 13452 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44325
I20260619 11:10:04.742384 13580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44325 every 8 connection(s)
I20260619 11:10:04.743320 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:04.752791 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13452
I20260619 11:10:04.752923 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:04.753229 7529 external_mini_cluster.cc:1428] Setting key 2a5bf301b8c73a43f34656e6920c7c3a
I20260619 11:10:04.757167 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.745218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:39615 (local address 127.7.90.126:40611)
0619 11:10:04.745374 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.745379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.746378 (+ 999us) server_negotiation.cc:407] Connection header received
0619 11:10:04.747669 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.747671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.747720 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.747802 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.748931 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.749473 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.750656 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.750864 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.750942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.753662 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.753665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.753667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.753690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.755277 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.755892 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.755895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.755896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.755976 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.756266 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.756268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.756269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.756433 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.756506 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.756959 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.757051 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:10:04.758042 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.745757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40611 (local address 127.7.90.67:39615)
0619 11:10:04.746218 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:04.746259 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:04.747473 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:04.747955 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:04.747962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:04.748299 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:04.748770 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.748782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.749666 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.749669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.750550 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:04.750558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.750993 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.750996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:04.751126 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.751757 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:04.751772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:04.753428 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:04.755389 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.755393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.755428 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.755801 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.756058 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:04.756061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:04.756062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:04.756187 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:04.756519 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:04.756524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:04.756744 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 11:10:04.756897 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 11:10:04.757151 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260619 11:10:04.759044 13581 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40611
I20260619 11:10:04.759312 13581 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:04.759804 13581 heartbeater.cc:511] Master 127.7.90.126:40611 requested a full tablet report, sending...
I20260619 11:10:04.760794 13122 ts_manager.cc:194] Registered new tserver with Master: 05fbfdca285b41cc8a1e01cf402bf97e (127.7.90.67:44325)
I20260619 11:10:04.761445 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:39615
I20260619 11:10:04.767517 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/test-admin.keytab.
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.784300 7529 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 11:10:04.787212 13588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:04.785785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.7.90.65:33761)
0619 11:10:04.786097 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.786117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.786141 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:10:04.786193 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.786198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.786385 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.786511 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.787006 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59048: BlockingRecv error: recv got EOF from 127.0.0.1:59048 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":120,"threads_started":1}
W20260619 11:10:04.787449 7529 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc)
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.788596 7529 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.789747 7529 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:10:04.800242 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.790770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.7.90.65:33761)
0619 11:10:04.790890 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.790893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.790910 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:10:04.791032 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.791035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.791074 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.791143 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.791621 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.792399 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.793492 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.793978 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.794079 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.796422 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.796432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.796447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.796475 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.798418 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.798839 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.798844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.798845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.798923 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.799279 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.799283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.799284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.799567 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.799731 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.799976 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.800110 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 11:10:04.804288 13588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:04.803229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.7.90.65:33761)
0619 11:10:04.803363 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.803368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.803512 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:10:04.803608 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.803611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.803653 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.803726 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.804177 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59074: BlockingRecv error: recv got EOF from 127.0.0.1:59074 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 11:10:04.804333 7529 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.805397 7529 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.806301 7529 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:10:04.815131 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.807159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.7.90.65:33761)
0619 11:10:04.807284 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.807288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.807379 (+ 91us) server_negotiation.cc:407] Connection header received
0619 11:10:04.807527 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.807530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.807573 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.807659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.808167 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.808701 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.809634 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.809862 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.809977 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.812133 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.812136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.812139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.812166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.813464 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.813964 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.813969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.813971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.814047 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.814326 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.814329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.814330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.814493 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.814623 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.814867 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.814977 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:10:04.817909 13588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:04.816910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.7.90.65:33761)
0619 11:10:04.817026 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.817030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.817138 (+ 108us) server_negotiation.cc:407] Connection header received
0619 11:10:04.817234 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.817237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.817276 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.817367 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.817794 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59104: BlockingRecv error: recv got EOF from 127.0.0.1:59104 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:10:04.818089 7529 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867375844525-7529-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.819097 7529 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:04.820024 7529 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:10:04.828140 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:04.820782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.7.90.65:33761)
0619 11:10:04.820894 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:04.820899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:04.821000 (+ 101us) server_negotiation.cc:407] Connection header received
0619 11:10:04.821086 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:04.821088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:04.821122 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:04.821182 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:04.821686 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.822157 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.822886 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:04.823081 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:04.823152 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:04.825301 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:04.825304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:04.825306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:04.825326 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:04.826661 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.827063 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.827065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.827067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.827132 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:04.827406 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:04.827408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:04.827409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:04.827561 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:04.827652 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:04.827846 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 11:10:04.827956 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:10:04.829188 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13184
I20260619 11:10:04.834650 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13318
I20260619 11:10:04.840096 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13452
I20260619 11:10:04.845434 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13090
2026-06-19T11:10:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2205 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13601](info): set up 2 sockets
Jun 19 11:10:04 dist-test-slave-f1t5 krb5kdc[13601](info): commencing operation
krb5kdc: starting...
W20260619 11:10:05.898051 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:05 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab.
Entry for principal oryx/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab.
I20260619 11:10:06.029282 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43695
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab
--principal=oryx/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:06.147015 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:06.147289 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:06.147329 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:06.151226 13614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:06.151275 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:06.151294 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:06.151310 13614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:06.151325 13614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:06.156558 13614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:43695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab
--principal=oryx/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:06.157569 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:06.158686 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:06.167485 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.167497 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.167538 13614 server_base.cc:1034] running on GCE node
W20260619 11:10:06.167495 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.168390 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:06.169656 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:06.170883 13614 hybrid_clock.cc:648] HybridClock initialized: now 1781867406170843 us; error 63 us; skew 500 ppm
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:06.173954 13614 init.cc:377] Logged in from keytab as oryx/127.7.90.126@KRBTEST.COM (short username oryx)
I20260619 11:10:06.174961 13614 webserver.cc:460] Webserver started at http://127.7.90.126:34325/ using document root <none> and password file <none>
I20260619 11:10:06.175566 13614 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:06.175670 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:06.175902 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:06.178066 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "0871ee46c95941c8a19ba0c00f0ff07b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "7db10c406c9674e9b67f302d65da0c46"
server_key_iv: "832e6ea74592b32a32f64c754026f8a8"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.178591 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "0871ee46c95941c8a19ba0c00f0ff07b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "7db10c406c9674e9b67f302d65da0c46"
server_key_iv: "832e6ea74592b32a32f64c754026f8a8"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.182377 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 11:10:06.184751 13629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.186169 13614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:06.186303 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "0871ee46c95941c8a19ba0c00f0ff07b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "7db10c406c9674e9b67f302d65da0c46"
server_key_iv: "832e6ea74592b32a32f64c754026f8a8"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.186414 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:06.216040 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:06.219883 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:06.220067 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:06.227849 13614 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:43695
I20260619 11:10:06.228183 13681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:43695 every 8 connection(s)
I20260619 11:10:06.228984 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:06.232121 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:06.237010 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13614
I20260619 11:10:06.237157 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:06.237473 7529 external_mini_cluster.cc:1428] Setting key 579b266a46bc5ec39c551a074ff0266c
I20260619 11:10:06.238654 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Bootstrap starting.
I20260619 11:10:06.241096 13682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:06.242101 13682 log.cc:824] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:06.244557 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: No bootstrap required, opened a new log
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:06.249504 13682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } }
I20260619 11:10:06.249876 13682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:06.249971 13682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0871ee46c95941c8a19ba0c00f0ff07b, State: Initialized, Role: FOLLOWER
I20260619 11:10:06.250464 13682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } }
I20260619 11:10:06.250628 13682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:06.250720 13682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:06.250828 13682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:06.252130 13682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } }
I20260619 11:10:06.252548 13682 leader_election.cc:302] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0871ee46c95941c8a19ba0c00f0ff07b; no voters:
I20260619 11:10:06.252935 13682 leader_election.cc:288] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:06.253140 13687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:06.253376 13687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 LEADER]: Becoming Leader. State: Replica: 0871ee46c95941c8a19ba0c00f0ff07b, State: Running, Role: LEADER
I20260619 11:10:06.253798 13687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } }
I20260619 11:10:06.254225 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.239087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.7.90.126:43695)
0619 11:10:06.239549 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:06.239572 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:06.239608 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:10:06.240279 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:06.240298 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:06.240648 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:06.240954 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:06.241955 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.243225 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.244471 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.245212 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.245351 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.248526 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:06.248532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:06.248550 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:06.248582 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:06.250596 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.251187 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.251194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.251200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.251348 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.251621 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.251625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.251626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.252235 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:06.252424 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:06.252815 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 11:10:06.253163 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":169,"threads_started":1}
I20260619 11:10:06.254437 13682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:06.255612 13689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0871ee46c95941c8a19ba0c00f0ff07b. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } }
I20260619 11:10:06.255734 13689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:06.256054 13688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } }
I20260619 11:10:06.256137 13688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:06.256961 13696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:06.260226 13696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:06.266019 13696 catalog_manager.cc:1347] Generated new cluster ID: 27dc78740afc4016bafa209025293415
I20260619 11:10:06.266117 13696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:06.272818 13696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:06.274202 13696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:06.281986 13696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Generated new TSK 0
I20260619 11:10:06.282884 13696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
I20260619 11:10:06.317800 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:06.434900 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:06.435206 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:06.435251 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:06.439504 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:06.439587 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:06.439652 13708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:06.445361 13708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:06.446617 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:06.447738 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:06.458636 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.458688 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.458657 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.458958 13708 server_base.cc:1034] running on GCE node
I20260619 11:10:06.459589 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:06.460316 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:06.461496 13708 hybrid_clock.cc:648] HybridClock initialized: now 1781867406461476 us; error 35 us; skew 500 ppm
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:06.464598 13708 init.cc:377] Logged in from keytab as oryx/127.7.90.65@KRBTEST.COM (short username oryx)
I20260619 11:10:06.465652 13708 webserver.cc:460] Webserver started at http://127.7.90.65:45441/ using document root <none> and password file <none>
I20260619 11:10:06.466262 13708 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:06.466326 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:06.466545 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:06.468699 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "c89ad2e93eac4ebe884db870d7509399"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "9eb93c1cf5e16f3ce8af4aa93c8bb3ae"
server_key_iv: "338a7a4634b400195b4a4d73a8439627"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.469298 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "c89ad2e93eac4ebe884db870d7509399"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "9eb93c1cf5e16f3ce8af4aa93c8bb3ae"
server_key_iv: "338a7a4634b400195b4a4d73a8439627"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.473412 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260619 11:10:06.476150 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.477494 13708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:06.477695 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "c89ad2e93eac4ebe884db870d7509399"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "9eb93c1cf5e16f3ce8af4aa93c8bb3ae"
server_key_iv: "338a7a4634b400195b4a4d73a8439627"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.477830 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:06.510202 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:06.514019 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:06.514274 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:06.514937 13708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:06.515820 13708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:06.515906 13708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.515981 13708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:06.516016 13708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.525684 13708 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:43085
I20260619 11:10:06.525712 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:43085 every 8 connection(s)
I20260619 11:10:06.526657 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:06.535578 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13708
I20260619 11:10:06.535744 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:06.536087 7529 external_mini_cluster.cc:1428] Setting key b4931636dfcb4516c285608316a19984
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:06.542403 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.528722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:58019 (local address 127.7.90.126:43695)
0619 11:10:06.528871 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:06.528875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:06.530101 (+ 1226us) server_negotiation.cc:407] Connection header received
0619 11:10:06.531006 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:06.531010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:06.531063 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:06.531148 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:06.532458 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.533008 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.534383 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.534617 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.534701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.537819 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:06.537822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:06.537825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:06.537853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:06.539797 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.540554 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.540556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.540557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.540644 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.541147 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.541149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.541150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.541338 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:06.541415 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:06.542162 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 11:10:06.542274 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:10:06.543718 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.529128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:58019)
0619 11:10:06.529931 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:06.529984 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:06.530771 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:06.531358 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:06.531366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:06.531768 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:06.532286 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.532300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.533157 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.533160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.534258 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.534268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.534763 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.534765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.534913 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.535602 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:06.535632 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:06.537625 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:06.539969 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.539978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.540003 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.540421 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.540805 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.540811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.540814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.541027 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.541464 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:06.541475 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:06.541882 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 11:10:06.542104 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 11:10:06.542538 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":122,"threads_started":1}
I20260619 11:10:06.545367 13837 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:06.545771 13837 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:06.546491 13837 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:10:06.548273 13646 ts_manager.cc:194] Registered new tserver with Master: c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:06.549489 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:58019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
I20260619 11:10:06.568863 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:06.681897 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:06.682272 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:06.682360 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:06.686466 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:06.686558 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:06.686651 13842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:06.692095 13842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:06.693253 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:06.694417 13842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:06.704279 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.704289 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.704308 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.704603 13842 server_base.cc:1034] running on GCE node
I20260619 11:10:06.705071 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:06.705777 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:06.706972 13842 hybrid_clock.cc:648] HybridClock initialized: now 1781867406706961 us; error 48 us; skew 500 ppm
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:06.709777 13842 init.cc:377] Logged in from keytab as oryx/127.7.90.66@KRBTEST.COM (short username oryx)
I20260619 11:10:06.710614 13842 webserver.cc:460] Webserver started at http://127.7.90.66:37459/ using document root <none> and password file <none>
I20260619 11:10:06.711141 13842 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:06.711197 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:06.711354 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:06.713342 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "7d40286feeb1440b8872f0fd085764fb"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "792ed3aa186eaa3551b027403f2b5d94"
server_key_iv: "6306f2e6f8c912571895b15120e6fd56"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.713968 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "7d40286feeb1440b8872f0fd085764fb"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "792ed3aa186eaa3551b027403f2b5d94"
server_key_iv: "6306f2e6f8c912571895b15120e6fd56"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.717626 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:06.719961 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.721197 13842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:06.721335 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "7d40286feeb1440b8872f0fd085764fb"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "792ed3aa186eaa3551b027403f2b5d94"
server_key_iv: "6306f2e6f8c912571895b15120e6fd56"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.721460 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:06.731616 13842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:06.734460 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:06.734675 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:06.735252 13842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:06.736049 13842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:06.736124 13842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.736188 13842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:06.736240 13842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.745319 13842 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45477
I20260619 11:10:06.745355 13970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45477 every 8 connection(s)
I20260619 11:10:06.746279 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:06.755950 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13842
I20260619 11:10:06.756131 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:06.756464 7529 external_mini_cluster.cc:1428] Setting key 5304f9803244801f7b9a0d6a150177be
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:06.761243 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.748101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:34733 (local address 127.7.90.126:43695)
0619 11:10:06.748228 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:06.748232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:06.750411 (+ 2179us) server_negotiation.cc:407] Connection header received
0619 11:10:06.751323 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:06.751326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:06.751400 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:06.751483 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:06.752644 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.753182 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.754396 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.754692 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.754808 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.757721 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:06.757726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:06.757729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:06.757759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:06.759232 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.759745 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.759749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.759751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.759833 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.760189 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.760192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.760194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.760372 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:06.760481 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:06.760966 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 11:10:06.761086 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:10:06.762140 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.748482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:34733)
0619 11:10:06.750261 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:06.750304 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:06.751110 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:06.751678 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:06.751685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:06.752022 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:06.752490 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.752503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.753311 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.753315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.754265 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.754274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.754844 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.754848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.755029 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.755733 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:06.755750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:06.757505 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:06.759354 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.759360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.759375 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.759635 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.759936 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.759939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.759941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.760085 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.760476 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:06.760483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:06.760726 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 11:10:06.760909 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 11:10:06.761239 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":689,"threads_started":1}
I20260619 11:10:06.763355 13971 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:06.763641 13971 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:06.764249 13971 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
I20260619 11:10:06.765437 13646 ts_manager.cc:194] Registered new tserver with Master: 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
WARNING: no policy specified for oryx/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:10:06.766345 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:34733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
I20260619 11:10:06.785485 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:06.894645 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:06.894919 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:06.894963 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:06.899008 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:06.899071 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:06.899179 13976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:06.904742 13976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:06.905933 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:06.906982 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:06.916649 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.916713 13976 server_base.cc:1034] running on GCE node
W20260619 11:10:06.916657 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:06.916667 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:06.917359 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:06.917984 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:06.919147 13976 hybrid_clock.cc:648] HybridClock initialized: now 1781867406919128 us; error 30 us; skew 500 ppm
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:06.921811 13976 init.cc:377] Logged in from keytab as oryx/127.7.90.67@KRBTEST.COM (short username oryx)
I20260619 11:10:06.922585 13976 webserver.cc:460] Webserver started at http://127.7.90.67:40463/ using document root <none> and password file <none>
I20260619 11:10:06.923084 13976 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:06.923141 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:06.923310 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:06.925330 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "362eaa563af8740972dc8f758dc5475e"
server_key_iv: "979745ba17e29c16d41ca56ad41b53cd"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.925904 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "362eaa563af8740972dc8f758dc5475e"
server_key_iv: "979745ba17e29c16d41ca56ad41b53cd"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.929746 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:10:06.932210 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.933737 13976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:06.933871 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "362eaa563af8740972dc8f758dc5475e"
server_key_iv: "979745ba17e29c16d41ca56ad41b53cd"
server_key_version: "encryptionkey@0"
I20260619 11:10:06.933990 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:06.944026 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:06.946998 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:06.947189 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:06.947722 13976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:06.948525 13976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:06.948596 13976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.948660 13976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:06.948693 13976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:06.957868 13976 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:42965
I20260619 11:10:06.957955 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:42965 every 8 connection(s)
I20260619 11:10:06.958811 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:06.961627 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 13976
I20260619 11:10:06.961752 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:06.962121 7529 external_mini_cluster.cc:1428] Setting key 1c04807c10d25e2358f6a55fa7ef6d74
Jun 19 11:10:06 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:06.972360 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.960764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:55679 (local address 127.7.90.126:43695)
0619 11:10:06.960911 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:06.960915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:06.962127 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 11:10:06.963122 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:06.963125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:06.963176 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:06.963252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:06.964573 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.965103 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.966294 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.966519 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.966664 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.968996 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:06.968999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:06.969002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:06.969024 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:06.970364 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.970856 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.970861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.970863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.970939 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:06.971324 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:06.971326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:06.971328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:06.971516 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:06.971622 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:06.972088 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 11:10:06.972205 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:10:06.973248 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:06.961160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:55679)
0619 11:10:06.961955 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:06.962014 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:06.962902 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:06.963411 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:06.963418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:06.963890 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:06.964431 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.964444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.965229 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.965233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.966170 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:06.966179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:06.966712 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:06.966716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:06.966882 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:06.967450 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:06.967466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:06.968827 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:06.970485 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.970491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.970505 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.970749 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.971045 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:06.971048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:06.971050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:06.971191 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:06.971629 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:06.971637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:06.971862 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 11:10:06.972051 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 11:10:06.972356 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":129,"threads_started":1}
I20260619 11:10:06.974429 14105 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:06.974722 14105 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:06.975286 14105 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
I20260619 11:10:06.976374 13646 ts_manager.cc:194] Registered new tserver with Master: 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965)
I20260619 11:10:06.976933 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:55679
I20260619 11:10:06.987977 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.90.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.125.keytab.
Entry for principal oryx/127.7.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.125.keytab.
I20260619 11:10:07.016326 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:46305
--webserver_interface=127.7.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.125.keytab
--principal=oryx/127.7.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:43695,127.7.90.125:46305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:07.126197 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:07.126508 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:07.126586 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:07.130672 14110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:07.130752 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:07.130787 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:07.130844 14110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:07.130882 14110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:07.136214 14110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:43695,127.7.90.125:46305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.125.keytab
--principal=oryx/127.7.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
W20260619 11:10:07.136475 14110 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.90.126:43695,127.7.90.125:46305'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 11:10:07.137465 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:07.138656 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.147496 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.147508 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.147564 14110 server_base.cc:1034] running on GCE node
W20260619 11:10:07.147472 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.148279 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.149359 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.150557 14110 hybrid_clock.cc:648] HybridClock initialized: now 1781867407150544 us; error 32 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.153223 14110 init.cc:377] Logged in from keytab as oryx/127.7.90.125@KRBTEST.COM (short username oryx)
I20260619 11:10:07.154158 14110 webserver.cc:460] Webserver started at http://127.7.90.125:36021/ using document root <none> and password file <none>
I20260619 11:10:07.154693 14110 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.154753 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.154915 14110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:07.157058 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/instance:
uuid: "d29c390c01b440c7a0539b339fa22d01"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "49740fd54c563ee24a31215455a7dcfc"
server_key_iv: "968658073aaa3c664b4b2ce01452917d"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.157665 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal/instance:
uuid: "d29c390c01b440c7a0539b339fa22d01"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "49740fd54c563ee24a31215455a7dcfc"
server_key_iv: "968658073aaa3c664b4b2ce01452917d"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.161834 14110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 11:10:07.164160 14125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.165504 14110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:07.165726 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "d29c390c01b440c7a0539b339fa22d01"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "49740fd54c563ee24a31215455a7dcfc"
server_key_iv: "968658073aaa3c664b4b2ce01452917d"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.165829 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.179347 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.182809 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:07.182976 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:07.190070 14110 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.125:46305
I20260619 11:10:07.190363 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.125:46305 every 8 connection(s)
I20260619 11:10:07.191202 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
I20260619 11:10:07.192524 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 14110
I20260619 11:10:07.194690 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:07.198629 14178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.125@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:07.208249 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.194227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.7.90.125:46305)
0619 11:10:07.194695 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.194710 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.194747 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:10:07.195458 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.195475 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.195797 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.196153 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.196960 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.198049 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.198929 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.199469 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.199640 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.202843 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.202852 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.202872 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.202910 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.205305 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.205891 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.205896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.205900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.206011 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.206325 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.206328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.206330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.206738 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.206971 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.207222 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.207495 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260619 11:10:07.210253 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.200451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.7.90.126:43695)
0619 11:10:07.200627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.200632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.201084 (+ 452us) server_negotiation.cc:407] Connection header received
0619 11:10:07.201228 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.201231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.201278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.201359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.202097 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.202750 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.203941 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.204285 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.204402 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.206807 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.206813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.206816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.206846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.208451 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.208909 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.208914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.208916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.209004 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.209329 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.209332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.209333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.209613 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.209716 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.210052 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.210143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:10:07.210355 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.200400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42670)
0619 11:10:07.200906 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.200976 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.201113 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.201449 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.201459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.201829 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.201962 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.201968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.202881 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.202884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.203812 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.203821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.204404 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.204407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.204499 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.204971 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.204982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.206662 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.208580 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.208587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.208591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.208796 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.209115 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.209123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.209125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.209226 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.209721 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.209727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.209843 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.209997 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.210165 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":193,"threads_started":1}
I20260619 11:10:07.211702 14178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.125@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
I20260619 11:10:07.219777 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.212244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43258)
0619 11:10:07.212371 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.212392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.212452 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.212649 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.212652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.212775 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.212861 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.212865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.214048 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.214052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.214665 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.214674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.215144 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.215147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.215217 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.215637 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.215647 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.216878 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.218591 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.218593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.218595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.218765 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.219014 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.219015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.219017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.219083 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.219388 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.219390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.219485 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.219564 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.219657 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 11:10:07.219950 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.212316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.7.90.125:46305)
0619 11:10:07.212436 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.212439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.212451 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:07.212526 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.212528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.212560 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.212623 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.212966 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.213937 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.214776 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.215021 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.215110 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.216993 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.216996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.216998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.217018 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.218501 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.218848 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.218850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.218851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.218935 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.219157 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.219160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.219160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.219312 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.219383 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.219647 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.219738 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:10:07.222683 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Bootstrap starting.
I20260619 11:10:07.224979 14178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:07.225870 14178 log.cc:824] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:07.227741 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: No bootstrap required, opened a new log
I20260619 11:10:07.230743 14178 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } }
I20260619 11:10:07.230984 14178 raft_consensus.cc:374] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:07.231071 14178 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29c390c01b440c7a0539b339fa22d01, State: Initialized, Role: FOLLOWER
I20260619 11:10:07.231505 14178 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } }
I20260619 11:10:07.232008 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } } }
I20260619 11:10:07.232137 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:07.232704 14178 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:10:07.237417 14195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:10:07.237598 14195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:10:07.239185 14110 master_runner.cc:428] Detected that this master d29c390c01b440c7a0539b339fa22d01 is joining an existing cluster
I20260619 11:10:07.239240 14110 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.90.125:46305
I20260619 11:10:07.249498 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.241793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.7.90.125:46305)
0619 11:10:07.241885 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.241889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.241915 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:10:07.242026 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.242029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.242072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.242153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.242701 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.243474 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.244324 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.244580 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.244695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.245986 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.245989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.245994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.246018 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.247855 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248298 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.248304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.248306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.248390 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248671 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.248674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.248676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.248892 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.248986 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.249212 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.249359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":38}
I20260619 11:10:07.249536 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.241281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43272)
0619 11:10:07.241819 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.241833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.241924 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.242197 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.242203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.242449 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.242602 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.242608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.243612 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.243618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.244205 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.244212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.244701 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.244704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.244794 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.245334 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.245348 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.245861 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.247983 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.247986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.247992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.248199 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.248493 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.248498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.248577 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.248965 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.248968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.249049 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.249171 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.249310 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":101,"threads_started":1}
I20260619 11:10:07.250829 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.241370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.7.90.126:43695)
0619 11:10:07.241705 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.241710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.242225 (+ 515us) server_negotiation.cc:407] Connection header received
0619 11:10:07.242400 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.242403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.242450 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.242529 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.243020 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.243516 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.244774 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.245132 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.245205 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.246530 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.246533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.246535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.246556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.248190 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248751 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.248755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.248756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.248828 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.249148 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.249154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.249158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.249365 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.249445 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.250554 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.250671 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260619 11:10:07.251276 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.241487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42676)
0619 11:10:07.242147 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.242162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.242281 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.242597 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.242600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.242811 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.242897 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.242901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.243638 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.243641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.244642 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.244652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.245350 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.245354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.245441 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.245947 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.245961 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.246396 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.248311 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.248317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.248600 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.248940 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.248943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.248945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.249037 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.249558 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.249561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.250355 (+ 794us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.250976 (+ 621us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.251106 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":30,"spinlock_wait_cycles":3456,"thread_start_us":82,"threads_started":1}
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.126@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
I20260619 11:10:07.266212 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.255894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43278)
0619 11:10:07.256691 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.256739 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.256996 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.257296 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.257302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.257762 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.257887 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.257894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.258584 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.258587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.259584 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.259594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.259995 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.259999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.260129 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.260601 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.260615 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.262399 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.264286 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.264297 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.264305 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.264639 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.264985 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.264988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.264989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.265089 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.265652 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.265659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.265775 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.265918 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.266077 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":88,"threads_started":1}
I20260619 11:10:07.266258 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.255945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.7.90.125:46305)
0619 11:10:07.256055 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.256060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.256822 (+ 762us) server_negotiation.cc:407] Connection header received
0619 11:10:07.257125 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.257128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.257170 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.257233 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.258008 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.258480 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.259699 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.259892 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.259963 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.262539 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.262544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.262548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.262573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.264165 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.264747 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.264751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.264753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.264845 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.265190 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.265193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.265195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.265446 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.265571 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.265995 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.266102 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:10:07.267894 13646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.90.125:46305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d29c390c01b440c7a0539b339fa22d01"
member_type: NON_VOTER
last_known_addr {
host: "127.7.90.125"
port: 46305
}
attrs {
promote: true
}
}
dest_uuid: "0871ee46c95941c8a19ba0c00f0ff07b"
cas_config_opid_index: -1
I20260619 11:10:07.268652 13646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.7.90.124@KRBTEST.COM; defaulting to no policy
I20260619 11:10:07.271359 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:07.270803 14206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d29c390c01b440c7a0539b339fa22d01 (127.7.90.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } }
I20260619 11:10:07.273793 14152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.124@KRBTEST.COM" created.
I20260619 11:10:07.274482 14206 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.90.125:46305
I20260619 11:10:07.274907 14110 master.cc:537] Master@127.7.90.125:46305 shutting down...
I20260619 11:10:07.278920 14205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:07.280149 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } }
I20260619 11:10:07.280270 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:07.280500 14110 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:07.280932 14110 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:07.281116 14110 tablet_replica.cc:330] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: stopping tablet replica
W20260619 11:10:07.282028 13630 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 11:10:07.284179 13630 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b -> Peer d29c390c01b440c7a0539b339fa22d01 (127.7.90.125:46305): Couldn't send request to peer d29c390c01b440c7a0539b339fa22d01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.124.keytab.
Entry for principal oryx/127.7.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.124.keytab.
I20260619 11:10:07.293326 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:38811
--webserver_interface=127.7.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.124.keytab
--principal=oryx/127.7.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:10:07.296736 14110 master.cc:552] Master@127.7.90.125:46305 shutdown complete.
I20260619 11:10:07.296801 14110 master_runner.cc:305] Clearing existing system tablet
I20260619 11:10:07.297076 14110 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.297137 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.298795 14110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:07.299901 14214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.300203 14110 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.300532 14110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:07.300629 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "d29c390c01b440c7a0539b339fa22d01"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "49740fd54c563ee24a31215455a7dcfc"
server_key_iv: "968658073aaa3c664b4b2ce01452917d"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.303723 14110 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:07.306521 14110 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:10:07.306679 14110 log.cc:1196] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 11:10:07.307001 14110 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Deleting consensus metadata
I20260619 11:10:07.307206 14110 master_runner.cc:315] Copying system tablet from 127.7.90.126:43695
I20260619 11:10:07.308415 14110 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: tablet copy: Beginning tablet copy session from remote peer at address 127.7.90.126:43695
I20260619 11:10:07.318684 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.309098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42690)
0619 11:10:07.309436 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.309452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.309617 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.309918 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.309922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.310150 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.310302 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.310310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.311464 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.311467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.312376 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.312395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.313012 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.313016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.313122 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.314343 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.314360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.314802 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.316869 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.316873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.316876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.317153 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.317487 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.317492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.317495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.317611 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.318090 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.318095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.318201 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.318333 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.318486 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260619 11:10:07.318766 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.309186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.7.90.126:43695)
0619 11:10:07.309311 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.309315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.309571 (+ 256us) server_negotiation.cc:407] Connection header received
0619 11:10:07.309723 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.309726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.309778 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.309877 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.310440 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.311309 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.312529 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.312881 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.312983 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.314961 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.314965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.314968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.314994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.316737 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.317269 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.317275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.317277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.317364 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.317720 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.317724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.317726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.317975 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.318088 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.318458 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.318590 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:10:07.319763 13666 tablet_copy_service.cc:140] P 0871ee46c95941c8a19ba0c00f0ff07b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d29c390c01b440c7a0539b339fa22d01 ({username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:42690)
I20260619 11:10:07.319948 13666 tablet_copy_service.cc:161] P 0871ee46c95941c8a19ba0c00f0ff07b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d29c390c01b440c7a0539b339fa22d01 at {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:42690: session id = d29c390c01b440c7a0539b339fa22d01-00000000000000000000000000000000
I20260619 11:10:07.321740 13666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:10:07.323365 14110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:10:07.325803 14110 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: tablet copy: Starting download of 0 data blocks...
I20260619 11:10:07.326001 14110 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: tablet copy: Starting download of 1 WAL segments...
I20260619 11:10:07.327701 14110 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:10:07.329588 13666 tablet_copy_service.cc:342] P 0871ee46c95941c8a19ba0c00f0ff07b: Request end of tablet copy session d29c390c01b440c7a0539b339fa22d01-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:42690
I20260619 11:10:07.329703 13666 tablet_copy_service.cc:433] P 0871ee46c95941c8a19ba0c00f0ff07b: ending tablet copy session d29c390c01b440c7a0539b339fa22d01-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d29c390c01b440c7a0539b339fa22d01
I20260619 11:10:07.337248 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.339116 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.339277 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.339494 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.339861 14110 server_base.cc:1034] running on GCE node
I20260619 11:10:07.340000 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.340305 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.341457 14110 hybrid_clock.cc:648] HybridClock initialized: now 1781867407341443 us; error 30 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.343080 14110 init.cc:377] Logged in from keytab as oryx/127.7.90.125@KRBTEST.COM (short username oryx)
I20260619 11:10:07.343534 14110 webserver.cc:460] Webserver started at http://127.7.90.125:37557/ using document root <none> and password file <none>
I20260619 11:10:07.343710 14110 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.343741 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.345000 14110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:07.346102 14231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.346356 14110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 11:10:07.346434 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "d29c390c01b440c7a0539b339fa22d01"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "49740fd54c563ee24a31215455a7dcfc"
server_key_iv: "968658073aaa3c664b4b2ce01452917d"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.346505 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.359690 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.361884 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:07.366410 14110 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.125:46305
I20260619 11:10:07.366444 14283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.125:46305 every 8 connection(s)
I20260619 11:10:07.366958 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
I20260619 11:10:07.368875 14284 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:10:07.370038 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Bootstrap starting.
I20260619 11:10:07.381075 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:10:07.381445 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Bootstrap complete.
I20260619 11:10:07.382102 14284 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } }
I20260619 11:10:07.382256 14284 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d29c390c01b440c7a0539b339fa22d01, State: Initialized, Role: LEARNER
I20260619 11:10:07.382375 14284 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } }
I20260619 11:10:07.382761 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } }
I20260619 11:10:07.382901 14284 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:07.382929 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:10:07.386370 14298 catalog_manager.cc:1259] Loaded cluster ID: 27dc78740afc4016bafa209025293415
I20260619 11:10:07.386498 14298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: loading cluster ID for follower catalog manager: success
I20260619 11:10:07.388682 14298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: acquiring CA information for follower catalog manager: success
I20260619 11:10:07.390213 14298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:10:07.426084 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:07.426335 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:07.426373 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:07.430408 14212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:07.430464 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:07.430482 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:07.430541 14212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:07.430562 14212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:07.435796 14212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.124.keytab
--principal=oryx/127.7.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:38811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:07.436892 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:07.438256 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.446851 14304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.446837 14306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.446832 14303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.447060 14212 server_base.cc:1034] running on GCE node
I20260619 11:10:07.447741 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.448781 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.449966 14212 hybrid_clock.cc:648] HybridClock initialized: now 1781867407449950 us; error 30 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.452633 14212 init.cc:377] Logged in from keytab as oryx/127.7.90.124@KRBTEST.COM (short username oryx)
I20260619 11:10:07.453475 14212 webserver.cc:460] Webserver started at http://127.7.90.124:41023/ using document root <none> and password file <none>
I20260619 11:10:07.454074 14212 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.454169 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.454428 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:07.456723 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/instance:
uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "8d2db837a23e4351af2ec5d64f99f82e"
server_key_iv: "3d85a9276fb469ea1ee886d618ee7432"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.457278 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal/instance:
uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "8d2db837a23e4351af2ec5d64f99f82e"
server_key_iv: "3d85a9276fb469ea1ee886d618ee7432"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.461000 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:07.463532 14313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.464784 14212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:07.464924 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "8d2db837a23e4351af2ec5d64f99f82e"
server_key_iv: "3d85a9276fb469ea1ee886d618ee7432"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.465041 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.493906 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.497713 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:07.497958 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:07.505769 14212 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.124:38811
I20260619 11:10:07.506139 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.124:38811 every 8 connection(s)
I20260619 11:10:07.506934 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
I20260619 11:10:07.509783 14366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:07.511066 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 14212
I20260619 11:10:07.514163 14366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.124@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:07.526332 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.516169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.7.90.126:43695)
0619 11:10:07.516291 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.516294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.516550 (+ 256us) server_negotiation.cc:407] Connection header received
0619 11:10:07.516675 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.516678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.516723 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.516801 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.517475 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.518052 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.519016 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.519325 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.519461 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.522576 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.522586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.522588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.522614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.524120 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.524598 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.524603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.524605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.524684 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.525036 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.525039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.525040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.525260 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.525372 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.525958 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.526086 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:10:07.527315 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.515991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42706)
0619 11:10:07.516410 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.516458 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.516578 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.516859 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.516864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.517203 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.517323 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.517329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.518219 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.518222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.518859 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.518876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.519524 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.519531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.519651 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.520172 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.520198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.522418 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.524239 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.524249 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.524258 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.524491 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.524787 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.524790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.524792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.524907 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.525368 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.525376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.525704 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.526588 (+ 884us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.526758 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":43,"thread_start_us":187,"threads_started":1}
I20260619 11:10:07.527323 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.512736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.7.90.124:38811)
0619 11:10:07.513189 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.513207 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.513242 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:10:07.514068 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.514091 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.514559 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.514860 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.515687 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.516964 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.517895 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.518780 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.518961 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.521866 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.521874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.521913 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.521945 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.524191 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.524674 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.524681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.524683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.524796 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.525199 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.525202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.525204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.525742 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.525969 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.526221 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.526505 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":6528,"thread_start_us":124,"threads_started":1}
I20260619 11:10:07.528584 14366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.124@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
I20260619 11:10:07.537633 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.529079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43284)
0619 11:10:07.529410 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.529426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.529498 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.530766 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.530768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.530906 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.530993 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.530998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.531903 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.531905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.532431 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.532438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.532875 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.532877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.532963 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.533358 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.533368 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.534643 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.536138 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.536141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.536142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.536311 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.536555 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.536557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.536558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.536657 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.537088 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.537092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.537177 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.537299 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.537429 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260619 11:10:07.537655 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.529190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.7.90.125:46305)
0619 11:10:07.530508 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.530511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.530525 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:10:07.530563 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.530565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.530606 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.530692 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.531134 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.531798 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.532538 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.532771 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.532873 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.534768 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.534770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.534772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.534794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.536044 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.536391 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.536394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.536395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.536471 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.536772 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.536778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.536785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.536972 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.537087 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.537349 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.537465 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":91,"threads_started":1}
I20260619 11:10:07.538429 14366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.124@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
I20260619 11:10:07.545876 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.538897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51764)
0619 11:10:07.539031 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.539044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.539108 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.539294 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.539296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.539410 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.539471 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.539475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.540501 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.540503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.540976 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.540982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.541368 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.541370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.541422 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.541789 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.541799 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.542911 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.544645 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.544647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.544648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.544829 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.545061 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.545063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.545065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.545136 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.545434 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.545436 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.545501 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.545657 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.545743 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 11:10:07.545895 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.538976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.7.90.124:38811)
0619 11:10:07.539088 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.539091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.539103 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:07.539172 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.539175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.539206 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.539266 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.539595 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.540408 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.541076 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.541279 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.541351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.543053 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.543063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.543065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.543083 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.544504 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.544913 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.544916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.544917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.544979 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.545213 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.545215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.545216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.545355 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.545426 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.545702 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.545785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:10:07.548880 14366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Bootstrap starting.
I20260619 11:10:07.551126 14366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:07.551817 14366 log.cc:824] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:07.552358 13837 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:07.553841 14366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: No bootstrap required, opened a new log
I20260619 11:10:07.556845 14366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } }
I20260619 11:10:07.557090 14366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:07.557170 14366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655b433fe5a944e3b7b93f258fe8f2e2, State: Initialized, Role: FOLLOWER
I20260619 11:10:07.557636 14366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } }
I20260619 11:10:07.558084 14372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } } }
I20260619 11:10:07.558190 14372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:07.558768 14366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:10:07.563310 14383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:10:07.563388 14383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:10:07.567438 14212 master_runner.cc:428] Detected that this master 655b433fe5a944e3b7b93f258fe8f2e2 is joining an existing cluster
I20260619 11:10:07.567525 14212 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.90.124:38811
I20260619 11:10:07.579175 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.569591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42710)
0619 11:10:07.569888 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.569905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.570003 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.570463 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.570466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.570658 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.570752 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.570757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.571748 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.571754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.572377 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.572384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.572869 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.572873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.572959 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.573547 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.573572 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.574039 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.577486 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.577490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.577492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.577778 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.578096 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.578100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.578102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.578202 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.578641 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.578644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.578733 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.578826 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.578976 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":768,"thread_start_us":102,"threads_started":1}
I20260619 11:10:07.579255 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.569994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.7.90.126:43695)
0619 11:10:07.570116 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.570120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.570133 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:07.570175 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.570178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.570221 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.570297 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.570876 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.571637 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.572502 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.572738 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.572863 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.574194 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.574198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.574202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.574231 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.577373 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.577886 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.577889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.577891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.577987 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.578299 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.578302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.578304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.578510 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.578588 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.578935 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.579073 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:07.580611 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13708
I20260619 11:10:07.584122 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.570372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.7.90.125:46305)
0619 11:10:07.570484 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.570489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.570501 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:07.570542 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.570545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.570591 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.571067 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.571574 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.572375 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.574461 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.574834 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.574956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.576070 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.576075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.576078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.576107 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.577900 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.578290 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.578295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.578297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.578378 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.578657 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.578662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.578664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.578872 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.578960 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.582031 (+ 3071us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.583904 (+ 1873us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:10:07.585052 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.569690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43298)
0619 11:10:07.570283 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.570297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.570363 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.571099 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.571102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.571319 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.571424 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.571430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.572974 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.572977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.574333 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.574340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.574964 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.574967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.575087 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.575452 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.575493 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.575943 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.577996 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.577999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.578001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.578188 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.578475 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.578478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.578480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.578565 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.579002 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.579005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.579112 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.584648 (+ 5536us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.584822 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":90,"threads_started":1}
I20260619 11:10:07.588192 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:43085
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:43695
--webserver_port=45441
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867406, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.126@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
I20260619 11:10:07.603412 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.571202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.7.90.124:38811)
0619 11:10:07.571305 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.571309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.579405 (+ 8096us) server_negotiation.cc:407] Connection header received
0619 11:10:07.579559 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.579563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.579613 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.579679 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.580147 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.580791 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.581770 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.582500 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.582573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.583747 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.583751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.583754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.583779 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.585577 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.586030 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.586034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.586036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.601922 (+ 15886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.602323 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.602328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.602331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.602611 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.602719 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.603038 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.603188 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:10:07.603978 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.570026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51776)
0619 11:10:07.579343 (+ 9317us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.579357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.579456 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.579768 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.579770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.579930 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.580028 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.580035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.580900 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.580903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.581653 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.581663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.582694 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.582697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.582775 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.583230 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.583243 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.583627 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.585688 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.585693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.585697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.585927 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.602086 (+ 16159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.602092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.602099 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.602209 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.602740 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.602743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.602835 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.603654 (+ 819us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.603786 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9260,"thread_start_us":87,"threads_started":1}
I20260619 11:10:07.606180 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.583541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51780)
0619 11:10:07.583734 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.583748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.583814 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.588147 (+ 4333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.588151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.588338 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.588471 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.588481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.589649 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.589654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.590310 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.590319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.590844 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.590849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.590947 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.591468 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.591481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.592740 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.604547 (+ 11807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.604549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.604551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.604792 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.605121 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.605124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.605126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.605223 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.605650 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.605655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.605751 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.605870 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.606026 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 11:10:07.606195 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.585904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.7.90.124:38811)
0619 11:10:07.587866 (+ 1962us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.587874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.587886 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:07.587961 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.587964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.588011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.588096 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.588627 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.589473 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.590444 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.590709 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.590826 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.592895 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.592899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.592901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.592926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.604367 (+ 11441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.604905 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.604912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.604915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.604999 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.605341 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.605344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.605345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.605505 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.605648 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.605918 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.606023 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1898,"thread_start_us":107,"threads_started":1}
I20260619 11:10:07.607347 13646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.90.124:38811: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
member_type: NON_VOTER
last_known_addr {
host: "127.7.90.124"
port: 38811
}
attrs {
promote: true
}
}
dest_uuid: "0871ee46c95941c8a19ba0c00f0ff07b"
cas_config_opid_index: 5
I20260619 11:10:07.607939 13646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } }
I20260619 11:10:07.609006 14207 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.610553 14340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:07.610849 14207 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.90.124:38811
I20260619 11:10:07.611033 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.611156 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:07.611572 14212 master.cc:537] Master@127.7.90.124:38811 shutting down...
I20260619 11:10:07.611958 14340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:10:07.612502 14206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:10:07.614506 13630 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b -> Peer 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124:38811): Couldn't send request to peer 655b433fe5a944e3b7b93f258fe8f2e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 11:10:07.614809 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.609293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43300)
0619 11:10:07.610364 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.610381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.610461 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.610921 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.610924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.610940 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:07.611031 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.611038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.612045 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.612049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.613200 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.613212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.614351 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.614358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.614457 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.614532 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.614637 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.614707 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003}
I20260619 11:10:07.615195 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.609524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.7.90.125:46305)
0619 11:10:07.610273 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.610278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.610542 (+ 264us) server_negotiation.cc:407] Connection header received
0619 11:10:07.610612 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.610618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.610689 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.610786 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:07.611157 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.611914 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.613333 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.614006 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.614102 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.614287 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.614985 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.615066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600}
I20260619 11:10:07.615633 14252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 LEARNER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:10:07.616513 14206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 11:10:07.616788 14212 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:07.617074 14212 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:07.617161 14212 tablet_replica.cc:330] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: stopping tablet replica
I20260619 11:10:07.618341 14252 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.620070 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0871ee46c95941c8a19ba0c00f0ff07b. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.620184 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:10:07.620102 14395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: NON_VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: true } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.620345 14395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:10:07.621455 14206 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: attempting to promote NON_VOTER d29c390c01b440c7a0539b339fa22d01 to VOTER
I20260619 11:10:07.621842 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } }
I20260619 11:10:07.623035 14252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 LEARNER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 11:10:07.623422 14209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 11:10:07.623528 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.623250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51794)
0619 11:10:07.623392 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.623461 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:38811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 11:10:07.623611 13630 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b -> Peer 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124:38811): Couldn't send request to peer 655b433fe5a944e3b7b93f258fe8f2e2. Status: Network error: Client connection negotiation failed: client connection to 127.7.90.124:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:10:07.624948 14205 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d29c390c01b440c7a0539b339fa22d01 (127.7.90.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.625411 14252 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d29c390c01b440c7a0539b339fa22d01 (127.7.90.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.626384 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.626477 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:07.626587 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.626650 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:07.627166 14395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.627272 14395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:07.634095 14212 master.cc:552] Master@127.7.90.124:38811 shutdown complete.
I20260619 11:10:07.634150 14212 master_runner.cc:305] Clearing existing system tablet
I20260619 11:10:07.634390 14212 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.634434 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.635737 14212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:07.636612 14397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.636801 14212 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.636911 14212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:07.637032 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "8d2db837a23e4351af2ec5d64f99f82e"
server_key_iv: "3d85a9276fb469ea1ee886d618ee7432"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.639144 14212 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:07.641331 14212 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:10:07.641414 14212 log.cc:1196] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 11:10:07.641680 14212 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Deleting consensus metadata
I20260619 11:10:07.641846 14212 master_runner.cc:315] Copying system tablet from 127.7.90.126:43695
I20260619 11:10:07.642897 14212 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: tablet copy: Beginning tablet copy session from remote peer at address 127.7.90.126:43695
I20260619 11:10:07.652292 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.643435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:42716)
0619 11:10:07.643829 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.643844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.643961 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.644249 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.644252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.644467 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.644584 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.644590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.645717 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.645721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.646654 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.646664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.647368 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.647371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.647476 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.648090 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.648110 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.648610 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.650698 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.650702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.650704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.650960 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.651267 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.651271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.651273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.651360 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.651760 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.651764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.651858 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.651971 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.652099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":153,"threads_started":1}
I20260619 11:10:07.652299 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.643541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.7.90.126:43695)
0619 11:10:07.643674 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.643678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.643928 (+ 250us) server_negotiation.cc:407] Connection header received
0619 11:10:07.644062 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.644065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.644116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.644210 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.644697 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.645506 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.646786 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.647137 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.647213 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.648784 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.648789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.648794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.648826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.650576 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.651074 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.651077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.651079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.651165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.651455 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.651458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.651459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.651658 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.651753 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.652038 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.652155 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:10:07.652966 13666 tablet_copy_service.cc:140] P 0871ee46c95941c8a19ba0c00f0ff07b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 655b433fe5a944e3b7b93f258fe8f2e2 ({username='oryx', principal='oryx/127.7.90.124@KRBTEST.COM'} at 127.0.0.1:42716)
I20260619 11:10:07.653065 13666 tablet_copy_service.cc:161] P 0871ee46c95941c8a19ba0c00f0ff07b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 655b433fe5a944e3b7b93f258fe8f2e2 at {username='oryx', principal='oryx/127.7.90.124@KRBTEST.COM'} at 127.0.0.1:42716: session id = 655b433fe5a944e3b7b93f258fe8f2e2-00000000000000000000000000000000
I20260619 11:10:07.654178 13666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:10:07.655438 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:10:07.657634 14212 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: tablet copy: Starting download of 0 data blocks...
I20260619 11:10:07.657811 14212 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: tablet copy: Starting download of 1 WAL segments...
I20260619 11:10:07.659300 14212 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:10:07.661170 13666 tablet_copy_service.cc:342] P 0871ee46c95941c8a19ba0c00f0ff07b: Request end of tablet copy session 655b433fe5a944e3b7b93f258fe8f2e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.90.124@KRBTEST.COM'} at 127.0.0.1:42716
I20260619 11:10:07.661250 13666 tablet_copy_service.cc:433] P 0871ee46c95941c8a19ba0c00f0ff07b: ending tablet copy session 655b433fe5a944e3b7b93f258fe8f2e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 655b433fe5a944e3b7b93f258fe8f2e2
I20260619 11:10:07.668414 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.670332 14407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.670405 14404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.670761 14212 server_base.cc:1034] running on GCE node
W20260619 11:10:07.673720 14405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.674011 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.674340 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.675494 14212 hybrid_clock.cc:648] HybridClock initialized: now 1781867407675470 us; error 45 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.677124 14212 init.cc:377] Logged in from keytab as oryx/127.7.90.124@KRBTEST.COM (short username oryx)
I20260619 11:10:07.677650 14212 webserver.cc:460] Webserver started at http://127.7.90.124:42103/ using document root <none> and password file <none>
I20260619 11:10:07.677841 14212 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.677882 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.679222 14212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:10:07.680235 14414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.680512 14212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:10:07.680583 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
format_stamp: "Formatted at 2026-06-19 11:10:07 on dist-test-slave-f1t5"
server_key: "8d2db837a23e4351af2ec5d64f99f82e"
server_key_iv: "3d85a9276fb469ea1ee886d618ee7432"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.680665 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.695166 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.697343 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:07.702383 14212 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.124:38811
I20260619 11:10:07.703007 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
I20260619 11:10:07.704816 14467 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:10:07.705699 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.124:38811 every 8 connection(s)
I20260619 11:10:07.706072 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Bootstrap starting.
I20260619 11:10:07.717551 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:10:07.718009 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: Bootstrap complete.
I20260619 11:10:07.718637 14467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } }
I20260619 11:10:07.718869 14467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 655b433fe5a944e3b7b93f258fe8f2e2, State: Initialized, Role: LEARNER
I20260619 11:10:07.719034 14467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } }
I20260619 11:10:07.719485 14470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:07.719631 14470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:10:07.719566 14467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:07.723111 14481 catalog_manager.cc:1259] Loaded cluster ID: 27dc78740afc4016bafa209025293415
I20260619 11:10:07.723209 14481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: loading cluster ID for follower catalog manager: success
I20260619 11:10:07.725780 14481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: acquiring CA information for follower catalog manager: success
I20260619 11:10:07.727229 14481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:10:07.737360 14393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:07.737716 14393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:07.737803 14393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:07.741986 14393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:07.742034 14393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:07.742126 14393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:07.747928 14393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:43085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=45441
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:07.748965 14393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:07.750025 14393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.760070 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.760030 14393 server_base.cc:1034] running on GCE node
W20260619 11:10:07.760011 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.760079 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.760682 14393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.761281 14393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.762477 14393 hybrid_clock.cc:648] HybridClock initialized: now 1781867407762463 us; error 26 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.765270 14393 init.cc:377] Logged in from keytab as oryx/127.7.90.65@KRBTEST.COM (short username oryx)
I20260619 11:10:07.766337 14393 webserver.cc:460] Webserver started at http://127.7.90.65:45441/ using document root <none> and password file <none>
I20260619 11:10:07.766938 14393 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.767053 14393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.768572 13971 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:07.771220 14393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:10:07.773312 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.774745 14393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:07.774909 14393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "c89ad2e93eac4ebe884db870d7509399"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "9eb93c1cf5e16f3ce8af4aa93c8bb3ae"
server_key_iv: "338a7a4634b400195b4a4d73a8439627"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.775487 14393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.787331 14393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.791196 14393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:07.791429 14393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:07.792117 14393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:07.793061 14393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:07.793111 14393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.793190 14393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:07.793226 14393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.802685 14393 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:43085
I20260619 11:10:07.802739 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:43085 every 8 connection(s)
I20260619 11:10:07.803766 14393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:07.806731 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 14393
I20260619 11:10:07.806952 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13842
I20260619 11:10:07.814814 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:45477
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:43695
--webserver_port=37459
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
I20260619 11:10:07.821642 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.805822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:46311 (local address 127.7.90.125:46305)
0619 11:10:07.805969 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.805975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.808066 (+ 2091us) server_negotiation.cc:407] Connection header received
0619 11:10:07.808169 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.808172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.808229 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.808306 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.811623 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.812429 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.813491 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.813888 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.813969 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.816932 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.816936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.816939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.816966 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.818775 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.819434 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.819439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.819440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.819552 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820116 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.820120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.820122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.820366 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.820464 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.821270 (+ 806us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.821423 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:10:07.822100 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.805746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:37527 (local address 127.7.90.126:43695)
0619 11:10:07.805905 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.805911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.808344 (+ 2433us) server_negotiation.cc:407] Connection header received
0619 11:10:07.808418 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.808424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.808489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.808591 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.810091 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.810609 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.812742 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.813097 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.813174 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.817415 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.817420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.817424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.817455 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.819834 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820777 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.820782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.820783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.820867 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.821170 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.821173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.821176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.821385 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.821462 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.821796 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.821928 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:10:07.822979 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.807989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:59911 (local address 127.7.90.124:38811)
0619 11:10:07.808890 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:07.808895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:07.808915 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:10:07.808971 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:07.808974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:07.809035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:07.809134 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:07.810097 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.811335 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.812861 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.813319 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.813445 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.817643 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:07.817649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:07.817654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:07.817688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:07.819680 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820326 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.820330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.820331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.820442 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820789 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:07.820795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:07.820800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:07.821075 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:07.821193 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:07.822569 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0619 11:10:07.822728 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":87,"threads_started":1}
I20260619 11:10:07.823230 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.807293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.7.90.65:59911)
0619 11:10:07.808299 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.808311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.808376 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.809254 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.809256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.809433 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.809919 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.809938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.811495 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.811502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.812700 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.812714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.813631 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.814208 (+ 577us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.815302 (+ 1094us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.815836 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.815858 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.817423 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.819831 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.819837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.819842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.820177 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.820527 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.820538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.820650 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.821248 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.821253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.821377 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.821613 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.821995 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":46,"thread_start_us":89,"threads_started":1}
I20260619 11:10:07.823625 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.806202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:37527)
0619 11:10:07.806679 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.806726 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.807680 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.808612 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.808619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.809145 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.809919 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.809938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.810764 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.810767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.812618 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.812624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.813360 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.813363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.813588 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.814282 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.814302 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.817251 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.819950 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.819953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.819955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.820665 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.820970 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.820973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.820975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.821053 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.821515 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.821517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.821614 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.822289 (+ 675us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.822438 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":60,"spinlock_wait_cycles":33920,"thread_start_us":148,"threads_started":1}
I20260619 11:10:07.824086 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:07.806202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.7.90.65:46311)
0619 11:10:07.807957 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:07.807966 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:07.808078 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:07.808610 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:07.808619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:07.810883 (+ 2264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:07.811457 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.811467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.812576 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.812579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.813373 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:07.813380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:07.814130 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:07.814134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:07.814274 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:07.814801 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:07.814815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:07.816721 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:07.818914 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.818921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.818937 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.819312 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.819817 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:07.819820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:07.819822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:07.819997 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:07.820570 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:07.820579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:07.821004 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 11:10:07.822133 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0619 11:10:07.822492 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1690,"mutex_wait_us":94,"spinlock_wait_cycles":8576,"thread_start_us":187,"threads_started":1}
I20260619 11:10:07.824677 14611 heartbeater.cc:347] Connected to a master server at 127.7.90.124:38811
I20260619 11:10:07.824702 14610 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:07.824872 14610 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:07.824875 14611 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:07.825490 14610 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
I20260619 11:10:07.825490 14611 heartbeater.cc:511] Master 127.7.90.124:38811 requested a full tablet report, sending...
I20260619 11:10:07.825579 14612 heartbeater.cc:347] Connected to a master server at 127.7.90.125:46305
I20260619 11:10:07.825665 14612 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:07.825843 14612 heartbeater.cc:511] Master 127.7.90.125:46305 requested a full tablet report, sending...
I20260619 11:10:07.826841 13646 ts_manager.cc:194] Re-registered known tserver with Master: c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:07.827248 14431 ts_manager.cc:194] Registered new tserver with Master: c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:07.827533 14244 ts_manager.cc:194] Registered new tserver with Master: c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:07.827564 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:37527
W20260619 11:10:07.936461 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:07.936789 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:07.936874 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:07.941056 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:07.941164 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:07.941264 14618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:07.946856 14618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:45477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=37459
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:07.948086 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:07.949132 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:07.959723 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.959735 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:07.960024 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:07.959997 14618 server_base.cc:1034] running on GCE node
I20260619 11:10:07.960530 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:07.961174 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:07.962357 14618 hybrid_clock.cc:648] HybridClock initialized: now 1781867407962329 us; error 43 us; skew 500 ppm
Jun 19 11:10:07 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:07.965008 14618 init.cc:377] Logged in from keytab as oryx/127.7.90.66@KRBTEST.COM (short username oryx)
I20260619 11:10:07.965996 14618 webserver.cc:460] Webserver started at http://127.7.90.66:37459/ using document root <none> and password file <none>
I20260619 11:10:07.966563 14618 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:07.966655 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:07.970335 14618 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:07.972379 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:07.973737 14618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:07.973884 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "7d40286feeb1440b8872f0fd085764fb"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "792ed3aa186eaa3551b027403f2b5d94"
server_key_iv: "6306f2e6f8c912571895b15120e6fd56"
server_key_version: "encryptionkey@0"
I20260619 11:10:07.974485 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:07.978897 14105 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:07.996016 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:07.999387 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:07.999601 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:08.000221 14618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:08.001099 14618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:08.001176 14618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:08.001243 14618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:08.001276 14618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:08.010427 14618 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45477
I20260619 11:10:08.010452 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45477 every 8 connection(s)
I20260619 11:10:08.011430 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:08.012296 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 14618
I20260619 11:10:08.012480 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13976
I20260619 11:10:08.020224 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:42965
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:43695
--webserver_port=40463
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867407, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:08.030845 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.013405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:38531 (local address 127.7.90.126:43695)
0619 11:10:08.013574 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.013582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.015256 (+ 1674us) server_negotiation.cc:407] Connection header received
0619 11:10:08.016221 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.016225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.016279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.016333 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.018455 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019291 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021302 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021658 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021724 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.026139 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.026146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.026151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.026187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.028352 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.028912 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.028915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.028918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029000 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029437 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.029440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.029442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029678 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.029762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.030562 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.030679 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:08.030845 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.013275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36949 (local address 127.7.90.125:46305)
0619 11:10:08.013996 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.014001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.015385 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 11:10:08.016222 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.016225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.016279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.016333 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.018462 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019278 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021253 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021624 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021710 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.025878 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.025884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.025890 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.025922 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.028197 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.028906 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.028910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.028912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029000 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029449 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.029452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.029453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029678 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.029762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.030562 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.030679 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633}
I20260619 11:10:08.030937 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.016601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:55157 (local address 127.7.90.124:38811)
0619 11:10:08.016731 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.016738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.017026 (+ 288us) server_negotiation.cc:407] Connection header received
0619 11:10:08.017209 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.017215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.017275 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.017330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.018475 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019368 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021259 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021733 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.026163 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.026170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.026175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.026211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.028620 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029180 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.029188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.029191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029284 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029665 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.029670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.029674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.029914 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.029996 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.030577 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.030722 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:10:08.032701 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.016252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.7.90.66:55157)
0619 11:10:08.016942 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.016957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.017100 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.017486 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.017491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.017739 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.018328 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.018342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019523 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.019529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.021122 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.021135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021990 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.022184 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.022952 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.022978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.025968 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.028757 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.028763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.028767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.029042 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.029409 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.029417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.029512 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.030092 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.030097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.030277 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.031152 (+ 875us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.031589 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":509,"spinlock_wait_cycles":23296,"thread_start_us":75,"threads_started":1}
I20260619 11:10:08.032701 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.013862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.7.90.66:36949)
0619 11:10:08.015269 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.015285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.015979 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.016734 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.016743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.017328 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.018284 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.018304 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019474 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.019477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.021116 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.021126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021906 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.022163 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.023036 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.023053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.025617 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.028336 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.028342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.028358 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.028790 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.029109 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.029114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.029321 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.029830 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.029837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.030284 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.031545 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.031652 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"mutex_wait_us":581,"spinlock_wait_cycles":140544,"thread_start_us":141,"threads_started":1}
I20260619 11:10:08.032701 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.013862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:38531)
0619 11:10:08.015048 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.015119 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.015979 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.016734 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.016743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.017328 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.018284 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.018304 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.019474 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.019477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.021186 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.021193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.021908 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.021910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.022161 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.023133 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.023152 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.025981 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.028460 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.028464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.028466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.028790 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.029109 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.029112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.029114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.029321 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.029830 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.029837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.030283 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.031062 (+ 779us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.031473 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":523,"spinlock_wait_cycles":90752,"thread_start_us":141,"threads_started":1}
I20260619 11:10:08.034020 14747 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:08.034075 14748 heartbeater.cc:347] Connected to a master server at 127.7.90.124:38811
I20260619 11:10:08.034271 14749 heartbeater.cc:347] Connected to a master server at 127.7.90.125:46305
I20260619 11:10:08.034250 14748 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.034368 14749 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.034250 14747 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.034838 14749 heartbeater.cc:511] Master 127.7.90.125:46305 requested a full tablet report, sending...
I20260619 11:10:08.034838 14748 heartbeater.cc:511] Master 127.7.90.124:38811 requested a full tablet report, sending...
I20260619 11:10:08.034969 14747 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
I20260619 11:10:08.035920 13646 ts_manager.cc:194] Re-registered known tserver with Master: 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
I20260619 11:10:08.036007 14244 ts_manager.cc:194] Registered new tserver with Master: 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
I20260619 11:10:08.036027 14431 ts_manager.cc:194] Registered new tserver with Master: 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
I20260619 11:10:08.036641 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:38531
I20260619 11:10:08.079735 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.076026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.7.90.124:38811)
0619 11:10:08.076179 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.076184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.076203 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:10:08.076330 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.076337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.076418 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.076528 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.076748 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.077539 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.078641 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.079212 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.079339 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.079510 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.079545 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.079610 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:10:08.079735 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.075927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51796)
0619 11:10:08.076089 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.076107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.076208 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.076542 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.076544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.076557 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.076642 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.076646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.077688 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.077691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.078509 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.078518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.079342 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.079346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.079426 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.079450 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.079517 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.079593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:10:08.080595 14441 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 11:10:08.080952 14470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0871ee46c95941c8a19ba0c00f0ff07b. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:08.081117 14470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: LEARNER
W20260619 11:10:08.139019 14754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:08.139271 14754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:08.139346 14754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:08.143381 14754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:08.143442 14754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:08.143500 14754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:08.149085 14754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:42965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=40463
--tserver_master_addrs=127.7.90.126:43695,127.7.90.125:46305,127.7.90.124:38811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:08.150174 14754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:08.151188 14754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:08.161360 14759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:08.161347 14762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:08.161422 14760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:08.161696 14754 server_base.cc:1034] running on GCE node
I20260619 11:10:08.162132 14754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:08.162725 14754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:08.163914 14754 hybrid_clock.cc:648] HybridClock initialized: now 1781867408163905 us; error 28 us; skew 500 ppm
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867408, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:08.166699 14754 init.cc:377] Logged in from keytab as oryx/127.7.90.67@KRBTEST.COM (short username oryx)
I20260619 11:10:08.167546 14754 webserver.cc:460] Webserver started at http://127.7.90.67:40463/ using document root <none> and password file <none>
I20260619 11:10:08.168124 14754 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:08.168223 14754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:08.171993 14754 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:08.174029 14769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:08.175251 14754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:08.175383 14754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b"
format_stamp: "Formatted at 2026-06-19 11:10:06 on dist-test-slave-f1t5"
server_key: "362eaa563af8740972dc8f758dc5475e"
server_key_iv: "979745ba17e29c16d41ca56ad41b53cd"
server_key_version: "encryptionkey@0"
I20260619 11:10:08.175920 14754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:08.210147 14754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:08.213564 14754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:08.213795 14754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:08.214401 14754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:08.215248 14754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:08.215324 14754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:08.215389 14754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:08.215438 14754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:08.224794 14754 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:42965
I20260619 11:10:08.224846 14882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:42965 every 8 connection(s)
I20260619 11:10:08.225904 14754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:08.227859 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 14754
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867408, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.124@KRBTEST.COM
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867408, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.125@KRBTEST.COM
Jun 19 11:10:08 dist-test-slave-f1t5 krb5kdc[13601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867408, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:08.245307 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.229682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:45085 (local address 127.7.90.124:38811)
0619 11:10:08.229805 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.229809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.231257 (+ 1448us) server_negotiation.cc:407] Connection header received
0619 11:10:08.231333 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.231336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.231387 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.231494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.233365 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234229 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.236053 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.236397 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.236486 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.240497 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.240503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.240508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.240541 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.242221 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.242951 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.242955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.242957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.243054 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.243529 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.243534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.243537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.243819 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.243899 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.244950 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.245098 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260619 11:10:08.246120 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.232319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.7.90.124:38811)
0619 11:10:08.234545 (+ 2226us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.234550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.235318 (+ 768us) server_negotiation.cc:407] Connection header received
0619 11:10:08.235446 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.235449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.235490 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.235548 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.236454 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.237315 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.239197 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.239821 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.239898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.242408 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.242413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.242415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.242440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.244098 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.244643 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.244647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.244648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.244732 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.245031 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.245034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.245036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.245257 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.245380 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.245795 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.245944 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":2161,"thread_start_us":83,"threads_started":1}
I20260619 11:10:08.247159 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.228014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:59779 (local address 127.7.90.126:43695)
0619 11:10:08.228170 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.228175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.230229 (+ 2054us) server_negotiation.cc:407] Connection header received
0619 11:10:08.232749 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.232752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.232800 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.232848 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.233382 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234238 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.237944 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.238274 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.238371 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.242213 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.242217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.242219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.242252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.244363 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.245788 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.245792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.245793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.245885 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.246225 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.246228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.246231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.246445 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.246532 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.246894 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.247013 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 11:10:08.247222 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.229546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.7.90.67:45085)
0619 11:10:08.231172 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.231184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.231260 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.231658 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.231667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.232319 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.233204 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.233224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234401 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.234405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.235924 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.235934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.236675 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.236680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.236871 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.237708 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.237729 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.240288 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.242356 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.242365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.242389 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.242806 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.243181 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.243185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.243187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.243402 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.244204 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.244211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.244618 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.245574 (+ 956us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.246030 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"mutex_wait_us":235,"spinlock_wait_cycles":2816,"thread_start_us":71,"threads_started":1}
I20260619 11:10:08.247702 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.228485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:59779)
0619 11:10:08.230121 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.230134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.230906 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.232973 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.232975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.233154 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.233264 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.233270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234400 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.234403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.236487 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.236496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.239268 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.239272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.239374 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.240070 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.240083 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.242027 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.245363 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.245366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.245368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.245664 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.246003 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.246006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.246008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.246113 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.246592 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.246596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.246696 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.247378 (+ 682us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.247522 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"mutex_wait_us":1098,"spinlock_wait_cycles":33280,"thread_start_us":129,"threads_started":1}
I20260619 11:10:08.248454 14884 heartbeater.cc:347] Connected to a master server at 127.7.90.124:38811
I20260619 11:10:08.248683 14883 heartbeater.cc:347] Connected to a master server at 127.7.90.126:43695
I20260619 11:10:08.248800 14883 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.249320 14883 heartbeater.cc:511] Master 127.7.90.126:43695 requested a full tablet report, sending...
I20260619 11:10:08.248701 14884 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.250010 14884 heartbeater.cc:511] Master 127.7.90.124:38811 requested a full tablet report, sending...
I20260619 11:10:08.250389 13646 ts_manager.cc:194] Re-registered known tserver with Master: 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965)
I20260619 11:10:08.250595 14431 ts_manager.cc:194] Registered new tserver with Master: 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965)
I20260619 11:10:08.251096 13646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:59779
I20260619 11:10:08.251502 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.229826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:45329 (local address 127.7.90.125:46305)
0619 11:10:08.229945 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.229950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.230012 (+ 62us) server_negotiation.cc:407] Connection header received
0619 11:10:08.231117 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.231121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.231174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.231269 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.233364 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234219 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.236694 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.237004 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.237093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.241734 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.241738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.241741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.241773 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.245739 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.246728 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.246731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.246733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.246821 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.247104 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.247107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.247109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.247315 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.248022 (+ 707us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.251215 (+ 3193us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.251345 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:10:08.252915 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.228485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.7.90.67:45329)
0619 11:10:08.229879 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.229931 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.230916 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.231840 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.231843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.232319 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:08.233204 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.233224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.234401 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.234405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.236587 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.236593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.237608 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.237612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.237717 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.238545 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:08.238565 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:08.241512 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:08.246114 (+ 4602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.246117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.246119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.246581 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.246926 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:08.246929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:08.246931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:08.247007 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:08.248091 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:08.248094 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:08.248205 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.252492 (+ 4287us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.252684 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":1251,"spinlock_wait_cycles":5632,"thread_start_us":138,"threads_started":1}
I20260619 11:10:08.253835 14885 heartbeater.cc:347] Connected to a master server at 127.7.90.125:46305
I20260619 11:10:08.253921 14885 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:08.254089 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.232049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.7.90.125:46305)
0619 11:10:08.239687 (+ 7638us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.239693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.242174 (+ 2481us) server_negotiation.cc:407] Connection header received
0619 11:10:08.242226 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.242229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.242294 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.243205 (+ 911us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.243323 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.244190 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.248144 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.248490 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.248955 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.250170 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.250174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.250177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.250200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.251919 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.252374 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.252377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.252379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.252458 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.252763 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.252766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.252768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.253081 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.253719 (+ 638us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.253794 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.253907 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7548,"thread_start_us":85,"threads_started":1}
I20260619 11:10:08.254096 14885 heartbeater.cc:511] Master 127.7.90.125:46305 requested a full tablet report, sending...
I20260619 11:10:08.254768 14242 ts_manager.cc:194] Registered new tserver with Master: 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965)
I20260619 11:10:08.257586 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.231684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.7.90.126:43695)
0619 11:10:08.243355 (+ 11671us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.243361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.243375 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:10:08.247647 (+ 4272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.247650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.247703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.249723 (+ 2020us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.249841 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.250670 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.251931 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.252202 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.252323 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.254334 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.254338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.254341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.254364 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.256013 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.256423 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.256426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.256427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.256495 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.256719 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.256721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.256722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.256916 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.257011 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.257237 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.257320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11534,"thread_start_us":67,"threads_started":1}
I20260619 11:10:08.260145 13646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:08.263124 13646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:08.265565 14902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: NON_VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: true } } }
I20260619 11:10:08.265699 14902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:08.268955 14901 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: attempting to promote NON_VOTER 655b433fe5a944e3b7b93f258fe8f2e2 to VOTER
I20260619 11:10:08.269238 14901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } }
I20260619 11:10:08.272647 14441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 LEARNER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:10:08.272781 14252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Refusing update from remote peer 0871ee46c95941c8a19ba0c00f0ff07b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:10:08.273041 14902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:10:08.273221 14206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:10:08.275231 14901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.276546 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.276589 14901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.276669 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:08.276794 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.276875 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:08.276690 14901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:08.277279 14252 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.277320 14441 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.280359 14470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.280498 14470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:08.280481 14912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:08.280607 14912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:08.285866 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.279336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45477 (local address 127.0.0.1:45318)
0619 11:10:08.280220 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.280235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.280368 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.281585 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.281588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.281606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.281770 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.281778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283122 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.283129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.284248 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.284257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285359 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.285365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.285465 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.285488 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.285599 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.285680 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":20,"thread_start_us":144,"threads_started":1}
I20260619 11:10:08.286124 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.278298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43085 (local address 127.0.0.1:33948)
0619 11:10:08.279686 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.279702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.279784 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.281635 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.281638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.281657 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.281746 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.281752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283447 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.283450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.284694 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.284703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285748 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.285754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.285846 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.285867 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.285917 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.285994 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":63}
I20260619 11:10:08.286206 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.280423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.7.90.67:42965)
0619 11:10:08.280900 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.280914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.281369 (+ 455us) server_negotiation.cc:407] Connection header received
0619 11:10:08.281573 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.281590 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.281782 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.281911 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.282252 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283400 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.284712 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285296 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.285404 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.285575 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.285937 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.286051 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":110,"threads_started":1}
I20260619 11:10:08.286500 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.280042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:42965 (local address 127.0.0.1:35772)
0619 11:10:08.281279 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.281292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.281389 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.282031 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.282034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.282050 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.282130 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.282136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283522 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.283524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.284578 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.284587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285630 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.285633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.285709 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.285729 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.285793 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.285846 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":24,"thread_start_us":101,"threads_started":1}
I20260619 11:10:08.286782 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.278595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.7.90.65:43085)
0619 11:10:08.281119 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.281134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.281168 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:10:08.281248 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.281254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.281455 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.281894 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.282030 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283269 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.284869 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285615 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.286368 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.286470 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.286509 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.286615 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2391,"thread_start_us":111,"threads_started":1}
I20260619 11:10:08.287127 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.279502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.7.90.66:45477)
0619 11:10:08.280886 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.280900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.280935 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:10:08.281016 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.281022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.281270 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.281441 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.282048 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.283009 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.284380 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.285207 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.285314 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.285475 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.286755 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.286982 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":98,"threads_started":1}
I20260619 11:10:08.289428 14817 tablet_service.cc:1467] Processing CreateTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f (DEFAULT_TABLE table=test-table [id=0bfa755fa8454a3b918b9f4dd77f0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:08.289338 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f (DEFAULT_TABLE table=test-table [id=0bfa755fa8454a3b918b9f4dd77f0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:08.289429 14681 tablet_service.cc:1467] Processing CreateTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f (DEFAULT_TABLE table=test-table [id=0bfa755fa8454a3b918b9f4dd77f0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:08.290480 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f66ce8ca78f4712bc641a838ca23a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:08.290481 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f66ce8ca78f4712bc641a838ca23a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:08.290481 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f66ce8ca78f4712bc641a838ca23a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:08.297286 14917 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Bootstrap starting.
I20260619 11:10:08.298781 14919 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Bootstrap starting.
I20260619 11:10:08.299643 14917 tablet_bootstrap.cc:654] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:08.300491 14917 log.cc:824] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:08.300724 14919 tablet_bootstrap.cc:654] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:08.301366 14919 log.cc:824] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:08.302103 14918 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Bootstrap starting.
I20260619 11:10:08.302448 14917 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: No bootstrap required, opened a new log
I20260619 11:10:08.302634 14917 ts_tablet_manager.cc:1397] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:10:08.303267 14919 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: No bootstrap required, opened a new log
I20260619 11:10:08.303468 14919 ts_tablet_manager.cc:1397] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:10:08.303987 14918 tablet_bootstrap.cc:654] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:08.304626 14918 log.cc:824] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:08.306495 14918 tablet_bootstrap.cc:492] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: No bootstrap required, opened a new log
I20260619 11:10:08.306672 14918 ts_tablet_manager.cc:1397] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:10:08.306916 14917 raft_consensus.cc:348] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.307206 14917 raft_consensus.cc:374] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:08.307277 14917 raft_consensus.cc:729] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89ad2e93eac4ebe884db870d7509399, State: Initialized, Role: FOLLOWER
I20260619 11:10:08.307824 14917 consensus_queue.cc:260] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.308539 14610 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:08.308507 14919 raft_consensus.cc:348] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.308749 14919 raft_consensus.cc:374] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:08.308804 14919 raft_consensus.cc:729] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40286feeb1440b8872f0fd085764fb, State: Initialized, Role: FOLLOWER
I20260619 11:10:08.309202 14919 consensus_queue.cc:260] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.309892 14917 ts_tablet_manager.cc:1428] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 11:10:08.310405 14747 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:08.311124 14919 ts_tablet_manager.cc:1428] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 11:10:08.315959 14918 raft_consensus.cc:348] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.316162 14918 raft_consensus.cc:374] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:08.316195 14918 raft_consensus.cc:729] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, State: Initialized, Role: FOLLOWER
I20260619 11:10:08.316565 14918 consensus_queue.cc:260] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.317107 14883 heartbeater.cc:503] Master 127.7.90.126:43695 was elected leader, sending a full tablet report...
I20260619 11:10:08.317457 14918 ts_tablet_manager.cc:1428] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260619 11:10:08.477257 14886 tablet.cc:2369] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:08.491250 14925 raft_consensus.cc:482] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:08.491441 14925 raft_consensus.cc:504] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.492590 14925 leader_election.cc:288] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085), 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
I20260619 11:10:08.496414 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.492888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:56167 (local address 127.7.90.65:43085)
0619 11:10:08.493000 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.493005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.493055 (+ 50us) server_negotiation.cc:407] Connection header received
0619 11:10:08.493233 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.493236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.493288 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.493372 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.493759 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.494266 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495412 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.495809 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.495992 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.496202 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.496285 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:10:08.496417 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.492983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:47353 (local address 127.7.90.66:45477)
0619 11:10:08.493111 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.493118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.493134 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:10:08.493304 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.493307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.493359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.493449 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:08.493746 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.494282 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495445 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.495816 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.496007 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.496251 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.496318 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:08.496417 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.492791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43085 (local address 127.7.90.67:56167)
0619 11:10:08.492981 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.492997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.493117 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.493392 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.493395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.493409 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.493602 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.493610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.494383 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.494388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.495268 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.495277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.495937 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.496048 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.496083 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.496145 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.496264 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":43}
I20260619 11:10:08.496419 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.492899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45477 (local address 127.7.90.67:47353)
0619 11:10:08.493078 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:08.493092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:08.493177 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:08.493457 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:08.493460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:08.493474 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:08.493612 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.493618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.494400 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.494403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.495277 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:08.495286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.495972 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.495975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:08.496093 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.496109 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:10:08.496215 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:10:08.496306 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":115}
I20260619 11:10:08.497141 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f66ce8ca78f4712bc641a838ca23a9f" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad2e93eac4ebe884db870d7509399" is_pre_election: true
I20260619 11:10:08.497143 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f66ce8ca78f4712bc641a838ca23a9f" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40286feeb1440b8872f0fd085764fb" is_pre_election: true
I20260619 11:10:08.497470 14701 raft_consensus.cc:2454] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 0.
I20260619 11:10:08.497470 14564 raft_consensus.cc:2454] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 0.
I20260619 11:10:08.498019 14772 leader_election.cc:302] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, c89ad2e93eac4ebe884db870d7509399; no voters:
I20260619 11:10:08.498293 14925 raft_consensus.cc:2781] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:08.498383 14925 raft_consensus.cc:482] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:08.498426 14925 raft_consensus.cc:3037] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:08.499461 14925 raft_consensus.cc:504] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.499821 14925 leader_election.cc:288] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 election: Requested vote from peers c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085), 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477)
I20260619 11:10:08.500188 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f66ce8ca78f4712bc641a838ca23a9f" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad2e93eac4ebe884db870d7509399"
I20260619 11:10:08.500236 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f66ce8ca78f4712bc641a838ca23a9f" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40286feeb1440b8872f0fd085764fb"
I20260619 11:10:08.500293 14564 raft_consensus.cc:3037] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:08.500385 14701 raft_consensus.cc:3037] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:08.501320 14564 raft_consensus.cc:2454] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 1.
I20260619 11:10:08.501346 14701 raft_consensus.cc:2454] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 1.
I20260619 11:10:08.501665 14772 leader_election.cc:302] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, c89ad2e93eac4ebe884db870d7509399; no voters:
I20260619 11:10:08.501859 14925 raft_consensus.cc:2781] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:08.502095 14925 raft_consensus.cc:686] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 LEADER]: Becoming Leader. State: Replica: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, State: Running, Role: LEADER
I20260619 11:10:08.502681 14925 consensus_queue.cc:237] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:08.506266 13646 catalog_manager.cc:5495] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67). New cstate: current_term: 1 leader_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } health_report { overall_health: HEALTHY } } }
W20260619 11:10:08.512874 14750 tablet.cc:2369] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:10:08.555709 14613 tablet.cc:2369] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:08.576020 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.572628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.7.90.67:42965)
0619 11:10:08.572759 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.572763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.572775 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:08.572848 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.572850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.572905 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.572994 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:08.573225 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.573765 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.574572 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.574786 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.574864 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.575182 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:08.575622 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:08.575713 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.575771 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.575821 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:10:08.582829 14701 raft_consensus.cc:1261] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Refusing update from remote peer 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:08.582829 14564 raft_consensus.cc:1261] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Refusing update from remote peer 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:08.583666 14926 consensus_queue.cc:1035] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:08.583931 14925 consensus_queue.cc:1035] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:08.594297 14931 mvcc.cc:205] Tried to move back new op lower bound from 7298528905547194368 to 7298528905229869056. Current Snapshot: MvccSnapshot[applied={T|T < 7298528905547194368}]
I20260619 11:10:08.604286 13646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:08.604465 13646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:08.608758 13646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0871ee46c95941c8a19ba0c00f0ff07b: Sending DeleteTablet for 3 replicas of tablet 9f66ce8ca78f4712bc641a838ca23a9f
I20260619 11:10:08.609586 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:08 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:33948
I20260619 11:10:08.609805 14681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:08 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:45318
I20260619 11:10:08.610419 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f66ce8ca78f4712bc641a838ca23a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:08 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:35772
I20260619 11:10:08.611088 14939 tablet_replica.cc:330] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: stopping tablet replica
I20260619 11:10:08.611449 14939 raft_consensus.cc:2229] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:08.611698 14940 tablet_replica.cc:330] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: stopping tablet replica
I20260619 11:10:08.611799 14939 raft_consensus.cc:2258] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:08.612152 14941 tablet_replica.cc:330] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: stopping tablet replica
I20260619 11:10:08.612221 14940 raft_consensus.cc:2229] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:08.612413 14941 raft_consensus.cc:2229] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:08.612499 14940 raft_consensus.cc:2258] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:08.612963 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 13614
I20260619 11:10:08.614112 14941 raft_consensus.cc:2258] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:08.614712 14940 ts_tablet_manager.cc:1905] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:08.615638 14939 ts_tablet_manager.cc:1905] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:08.617274 14940 ts_tablet_manager.cc:1918] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:08.617360 14940 log.cc:1196] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/9f66ce8ca78f4712bc641a838ca23a9f
I20260619 11:10:08.617707 14940 ts_tablet_manager.cc:1939] T 9f66ce8ca78f4712bc641a838ca23a9f P 7d40286feeb1440b8872f0fd085764fb: Deleting consensus metadata
I20260619 11:10:08.618350 14939 ts_tablet_manager.cc:1918] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:08.618448 14939 log.cc:1196] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/9f66ce8ca78f4712bc641a838ca23a9f
I20260619 11:10:08.618813 14939 ts_tablet_manager.cc:1939] T 9f66ce8ca78f4712bc641a838ca23a9f P c89ad2e93eac4ebe884db870d7509399: Deleting consensus metadata
W20260619 11:10:08.620275 14634 connection.cc:535] server connection from 127.0.0.1:45318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:10:08.622282 14941 ts_tablet_manager.cc:1905] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:08.624058 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.611885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.7.90.125:46305)
0619 11:10:08.612001 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.612006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.613034 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 11:10:08.613596 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.613602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.613684 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.613741 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.614989 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.615729 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.617028 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.617374 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.617452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.620219 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.620222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.620225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.620252 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.622093 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.622644 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.622648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.622650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.622735 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.623053 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.623057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.623059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.623303 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.623408 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.623768 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.623897 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:10:08.624853 14941 ts_tablet_manager.cc:1918] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:08.624939 14941 log.cc:1196] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/9f66ce8ca78f4712bc641a838ca23a9f
I20260619 11:10:08.625252 14941 ts_tablet_manager.cc:1939] T 9f66ce8ca78f4712bc641a838ca23a9f P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting consensus metadata
W20260619 11:10:08.626233 14773 connection.cc:441] server connection from 127.0.0.1:35772 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35772 (request call id 1) could send its response
I20260619 11:10:08.633774 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.612401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.7.90.124:38811)
0619 11:10:08.612520 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.612525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.624686 (+ 12161us) server_negotiation.cc:407] Connection header received
0619 11:10:08.624863 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.624866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.624916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.624978 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.625990 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.626790 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.627613 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.627943 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.628059 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.629698 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.629702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.629705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.629730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.631601 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.632041 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.632044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.632046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.632129 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.632479 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.632482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.632485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.632701 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.632840 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.633429 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.633589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:10:08.635748 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.626250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.7.90.125:46305)
0619 11:10:08.626342 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.626345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.626635 (+ 290us) server_negotiation.cc:407] Connection header received
0619 11:10:08.626803 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.626806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.626844 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.626890 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.627797 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.628559 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.629758 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.630102 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.630169 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.631711 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.631714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.631717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.631740 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.633902 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.634371 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.634375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.634376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.634474 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.634821 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.634823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.634825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.635078 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.635209 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.635467 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.635577 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:10:08.637233 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:08.626715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.7.90.124:38811)
0619 11:10:08.626814 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:08.626820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:08.628071 (+ 1251us) server_negotiation.cc:407] Connection header received
0619 11:10:08.628226 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:08.628229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:08.628268 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:08.628316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:08.629344 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.630095 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.631361 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:08.631713 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:08.631785 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:08.633278 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:08.633283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:08.633287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:08.633314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:08.635531 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.635998 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.636003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.636005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.636118 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:08.636419 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:08.636422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:08.636424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:08.636650 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:08.636767 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:08.637011 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 11:10:08.637092 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:10:09.595976 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:09.595228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:51117)
0619 11:10:09.595748 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:09.595854 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":119,"threads_started":1}
W20260619 11:10:09.596609 14747 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:43695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:09.596863 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:09.596423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:43487)
0619 11:10:09.596691 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:09.596771 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
W20260619 11:10:09.597191 14610 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:43695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:09.600211 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:09.599868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:44305)
0619 11:10:09.600071 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:09.600135 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
W20260619 11:10:09.600462 14883 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:43695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.109892 14952 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0871ee46c95941c8a19ba0c00f0ff07b)
I20260619 11:10:10.110138 14952 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } }
I20260619 11:10:10.110989 14953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0871ee46c95941c8a19ba0c00f0ff07b)
I20260619 11:10:10.111169 14953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } }
I20260619 11:10:10.111362 14952 leader_election.cc:288] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695), 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124:38811)
I20260619 11:10:10.112032 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.111484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43170)
0619 11:10:10.111844 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.111927 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":63,"threads_started":1}
W20260619 11:10:10.112280 14233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.112584 14953 leader_election.cc:288] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695), d29c390c01b440c7a0539b339fa22d01 (127.7.90.125:46305)
I20260619 11:10:10.112763 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.112270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43186)
0619 11:10:10.112587 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.112655 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":68,"threads_started":1}
W20260619 11:10:10.112914 14416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.113343 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.113033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43198)
0619 11:10:10.113227 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.113284 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260619 11:10:10.113467 14233 leader_election.cc:334] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695): Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.114174 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.113948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43212)
0619 11:10:10.114064 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.114115 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 11:10:10.114271 14416 leader_election.cc:334] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695): Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.115906 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.111594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:38811 (local address 127.0.0.1:51830)
0619 11:10:10.111953 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.111969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:10.112084 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:10.112489 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:10.112492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:10.112511 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:10.112622 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.112628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.113701 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.113705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.114867 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.114877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.115511 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.115514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.115600 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.115627 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:10:10.115711 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 11:10:10.115795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
I20260619 11:10:10.115976 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.111760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.7.90.124:38811)
0619 11:10:10.111960 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.111965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.112031 (+ 66us) server_negotiation.cc:407] Connection header received
0619 11:10:10.112214 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.112217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.112277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.112357 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:10.112751 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.113567 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.114994 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.115390 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.115468 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.115511 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.115768 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.115836 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260619 11:10:10.116411 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d29c390c01b440c7a0539b339fa22d01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" is_pre_election: true
I20260619 11:10:10.116621 14441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d29c390c01b440c7a0539b339fa22d01 in term 1.
I20260619 11:10:10.116981 14957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.112360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:46305 (local address 127.0.0.1:43358)
0619 11:10:10.113123 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.113136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:10.113220 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:10.113558 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:10.113560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:10.113576 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:10.113696 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.113702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.114681 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.114685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.115645 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.115656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.116554 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.116558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.116656 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.116675 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:10:10.116770 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 11:10:10.116858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":76,"threads_started":1}
I20260619 11:10:10.117039 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.112602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.7.90.125:46305)
0619 11:10:10.113000 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.113008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.113291 (+ 283us) server_negotiation.cc:407] Connection header received
0619 11:10:10.113328 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.113330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.113369 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.113445 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:10.113819 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.114569 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.115768 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.116411 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.116527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.116590 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.116854 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.116925 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":110,"threads_started":1}
I20260619 11:10:10.116986 14232 leader_election.cc:302] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 655b433fe5a944e3b7b93f258fe8f2e2, d29c390c01b440c7a0539b339fa22d01; no voters: 0871ee46c95941c8a19ba0c00f0ff07b
I20260619 11:10:10.117336 14952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 11:10:10.117399 14952 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0871ee46c95941c8a19ba0c00f0ff07b)
I20260619 11:10:10.117475 14952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:10:10.117408 14252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d29c390c01b440c7a0539b339fa22d01" is_pre_election: true
I20260619 11:10:10.118654 14952 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } }
I20260619 11:10:10.118927 14252 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 655b433fe5a944e3b7b93f258fe8f2e2 in current term 2: Already voted for candidate d29c390c01b440c7a0539b339fa22d01 in this term.
I20260619 11:10:10.119102 14952 leader_election.cc:288] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 election: Requested vote from peers 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695), 655b433fe5a944e3b7b93f258fe8f2e2 (127.7.90.124:38811)
I20260619 11:10:10.119374 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.119188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43224)
0619 11:10:10.119268 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.119314 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 11:10:10.119617 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d29c390c01b440c7a0539b339fa22d01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "655b433fe5a944e3b7b93f258fe8f2e2"
I20260619 11:10:10.119756 14441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:10:10.120014 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.119803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43232)
0619 11:10:10.119901 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.119956 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 11:10:10.120121 14415 leader_election.cc:302] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655b433fe5a944e3b7b93f258fe8f2e2; no voters: 0871ee46c95941c8a19ba0c00f0ff07b, d29c390c01b440c7a0539b339fa22d01
W20260619 11:10:10.120494 14233 leader_election.cc:334] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695): Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
I20260619 11:10:10.121093 14441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d29c390c01b440c7a0539b339fa22d01 in term 2.
I20260619 11:10:10.121201 14953 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:10:10.121400 14232 leader_election.cc:302] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 655b433fe5a944e3b7b93f258fe8f2e2, d29c390c01b440c7a0539b339fa22d01; no voters: 0871ee46c95941c8a19ba0c00f0ff07b
I20260619 11:10:10.121654 14952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:10:10.121882 14952 raft_consensus.cc:686] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [term 2 LEADER]: Becoming Leader. State: Replica: d29c390c01b440c7a0539b339fa22d01, State: Running, Role: LEADER
I20260619 11:10:10.122159 14952 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } }
I20260619 11:10:10.123247 14962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29c390c01b440c7a0539b339fa22d01. Latest consensus state: current_term: 2 leader_uuid: "d29c390c01b440c7a0539b339fa22d01" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:10.123335 14962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:10.123608 14964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:10.125708 14964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:10.126096 14964 catalog_manager.cc:1259] Loaded cluster ID: 27dc78740afc4016bafa209025293415
I20260619 11:10:10.126170 14964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:10.126706 14964 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:10.127092 14964 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Loaded TSK: 0
I20260619 11:10:10.127718 14964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:10.152036 14242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:10.155206 14242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:10.157895 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.157620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43254)
0619 11:10:10.157750 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.157805 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:10:10.157955 14441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [term 2 FOLLOWER]: Refusing update from remote peer d29c390c01b440c7a0539b339fa22d01: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 11:10:10.158385 14962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 11:10:10.158581 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.158384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.0.0.1:43268)
0619 11:10:10.158471 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.158524 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 11:10:10.158704 14233 consensus_peers.cc:487] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 -> Peer 0871ee46c95941c8a19ba0c00f0ff07b (127.7.90.126:43695): Couldn't send request to peer 0871ee46c95941c8a19ba0c00f0ff07b. Status: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:10:10.159978 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29c390c01b440c7a0539b339fa22d01. Latest consensus state: current_term: 2 leader_uuid: "d29c390c01b440c7a0539b339fa22d01" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:10.160085 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:10.161103 14952 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d29c390c01b440c7a0539b339fa22d01" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:10.161208 14952 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:10.161617 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d29c390c01b440c7a0539b339fa22d01" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0871ee46c95941c8a19ba0c00f0ff07b" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 43695 } } peers { permanent_uuid: "d29c390c01b440c7a0539b339fa22d01" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 46305 } attrs { promote: false } } peers { permanent_uuid: "655b433fe5a944e3b7b93f258fe8f2e2" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 38811 } attrs { promote: false } } }
I20260619 11:10:10.161729 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655b433fe5a944e3b7b93f258fe8f2e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:10:10.171038 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.166826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43085 (local address 127.0.0.1:33960)
0619 11:10:10.166906 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.166945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:10.167018 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:10.167539 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:10.167545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:10.167567 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:10.167732 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.167739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.168762 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.168768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.169787 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.169797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.170670 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.170674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.170778 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.170802 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:10:10.170847 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 11:10:10.170911 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":25,"spinlock_wait_cycles":1280}
I20260619 11:10:10.171259 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.167072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:42965 (local address 127.0.0.1:35796)
0619 11:10:10.167468 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.167478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:10.167603 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:10.167881 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:10.167883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:10.167897 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:10.168002 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.168014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.168861 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.168864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.169938 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.169946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.170605 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.170609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.170722 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.170743 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:10:10.171019 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 11:10:10.171102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":22,"spinlock_wait_cycles":384,"thread_start_us":86,"threads_started":1}
I20260619 11:10:10.171608 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.167067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.7.90.65:43085)
0619 11:10:10.167295 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.167299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.167313 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:10:10.167365 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.167369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.167415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.167845 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:10.167973 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.168638 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.169920 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.170532 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.171303 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.171363 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.171402 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.171476 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260619 11:10:10.171664 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.167178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.7.90.67:42965)
0619 11:10:10.167373 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.167376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.167675 (+ 299us) server_negotiation.cc:407] Connection header received
0619 11:10:10.167738 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.167741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.167783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.167876 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:10.168129 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.168738 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.170072 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.170485 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.170610 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.170673 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.171448 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.171530 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":53,"threads_started":1}
I20260619 11:10:10.171792 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.166522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45477 (local address 127.0.0.1:45326)
0619 11:10:10.166630 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.166642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:10.166740 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:10.167594 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:10.167597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:10.167610 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:10.167739 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.167744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.169465 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.169469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.170597 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:10.170609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.171452 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.171455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:10.171529 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.171551 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:10:10.171610 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 11:10:10.171679 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":30}
I20260619 11:10:10.172096 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 (DEFAULT_TABLE table=test-table [id=b17f7de2a100468285e35ce2d234b3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:10.172197 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.166766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.7.90.66:45477)
0619 11:10:10.167340 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.167344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.167357 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:10.167400 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.167402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.167468 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.167585 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:10.167857 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.169321 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.170723 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.171334 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.171966 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.172024 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.172053 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.172105 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":87,"threads_started":1}
I20260619 11:10:10.172313 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e82f1fe95e94146ba94bd41acfff393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:10.172657 14681 tablet_service.cc:1467] Processing CreateTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 (DEFAULT_TABLE table=test-table [id=b17f7de2a100468285e35ce2d234b3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:10.172853 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e82f1fe95e94146ba94bd41acfff393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:10.173851 14817 tablet_service.cc:1467] Processing CreateTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 (DEFAULT_TABLE table=test-table [id=b17f7de2a100468285e35ce2d234b3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:10.174105 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e82f1fe95e94146ba94bd41acfff393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:10.174338 14975 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: Bootstrap starting.
I20260619 11:10:10.175081 14976 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Bootstrap starting.
I20260619 11:10:10.175252 14975 tablet_bootstrap.cc:654] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:10.175958 14976 tablet_bootstrap.cc:654] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:10.176535 14978 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Bootstrap starting.
I20260619 11:10:10.176640 14975 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: No bootstrap required, opened a new log
I20260619 11:10:10.176792 14975 ts_tablet_manager.cc:1397] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:10.177156 14975 raft_consensus.cc:348] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.177265 14975 raft_consensus.cc:374] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:10.177294 14975 raft_consensus.cc:729] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89ad2e93eac4ebe884db870d7509399, State: Initialized, Role: FOLLOWER
I20260619 11:10:10.177457 14978 tablet_bootstrap.cc:654] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:10.177436 14975 consensus_queue.cc:260] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.177774 14975 ts_tablet_manager.cc:1428] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:10.178512 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.178034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:40677)
0619 11:10:10.178344 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.178405 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260619 11:10:10.180326 14612 heartbeater.cc:503] Master 127.7.90.125:46305 was elected leader, sending a full tablet report...
I20260619 11:10:10.180684 14976 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: No bootstrap required, opened a new log
I20260619 11:10:10.180778 14976 ts_tablet_manager.cc:1397] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 11:10:10.181151 14976 raft_consensus.cc:348] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.181300 14976 raft_consensus.cc:374] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:10.181336 14976 raft_consensus.cc:729] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40286feeb1440b8872f0fd085764fb, State: Initialized, Role: FOLLOWER
I20260619 11:10:10.181471 14976 consensus_queue.cc:260] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.181829 14976 ts_tablet_manager.cc:1428] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:10.182231 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.181902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:57043)
0619 11:10:10.182094 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.182149 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 11:10:10.182945 14978 tablet_bootstrap.cc:492] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: No bootstrap required, opened a new log
I20260619 11:10:10.183087 14978 ts_tablet_manager.cc:1397] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 11:10:10.183566 14749 heartbeater.cc:503] Master 127.7.90.125:46305 was elected leader, sending a full tablet report...
I20260619 11:10:10.183594 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.182062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:55271)
0619 11:10:10.183445 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.183502 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1308,"thread_start_us":85,"threads_started":1}
I20260619 11:10:10.183524 14978 raft_consensus.cc:348] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.183642 14978 raft_consensus.cc:374] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:10.183684 14978 raft_consensus.cc:729] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, State: Initialized, Role: FOLLOWER
I20260619 11:10:10.183837 14978 consensus_queue.cc:260] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.184208 14978 ts_tablet_manager.cc:1428] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:10.184756 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.184225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:46689)
0619 11:10:10.184636 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.184694 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
I20260619 11:10:10.185211 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.184364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:43815)
0619 11:10:10.185073 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.185125 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":91,"threads_started":1}
I20260619 11:10:10.186028 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.185620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:44093)
0619 11:10:10.185880 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.185942 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":130,"threads_started":1}
I20260619 11:10:10.186385 14885 heartbeater.cc:503] Master 127.7.90.125:46305 was elected leader, sending a full tablet report...
I20260619 11:10:10.210155 14985 raft_consensus.cc:482] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:10.210310 14985 raft_consensus.cc:504] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.210776 14985 leader_election.cc:288] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477), c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:10.211176 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e82f1fe95e94146ba94bd41acfff393" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40286feeb1440b8872f0fd085764fb" is_pre_election: true
I20260619 11:10:10.211194 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e82f1fe95e94146ba94bd41acfff393" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad2e93eac4ebe884db870d7509399" is_pre_election: true
I20260619 11:10:10.211331 14701 raft_consensus.cc:2454] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 0.
I20260619 11:10:10.211354 14564 raft_consensus.cc:2454] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 0.
I20260619 11:10:10.211637 14772 leader_election.cc:302] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, c89ad2e93eac4ebe884db870d7509399; no voters:
I20260619 11:10:10.211804 14985 raft_consensus.cc:2781] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:10.211856 14985 raft_consensus.cc:482] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:10.211882 14985 raft_consensus.cc:3037] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:10.212834 14985 raft_consensus.cc:504] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.213150 14985 leader_election.cc:288] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 election: Requested vote from peers 7d40286feeb1440b8872f0fd085764fb (127.7.90.66:45477), c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085)
I20260619 11:10:10.213457 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e82f1fe95e94146ba94bd41acfff393" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40286feeb1440b8872f0fd085764fb"
I20260619 11:10:10.213589 14701 raft_consensus.cc:3037] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:10.213591 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e82f1fe95e94146ba94bd41acfff393" candidate_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad2e93eac4ebe884db870d7509399"
I20260619 11:10:10.213732 14564 raft_consensus.cc:3037] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:10.214481 14701 raft_consensus.cc:2454] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 1.
I20260619 11:10:10.214649 14564 raft_consensus.cc:2454] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19795b4f8ef54fd6bcb9bb0c35d0ab7b in term 1.
I20260619 11:10:10.214733 14773 leader_election.cc:302] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, 7d40286feeb1440b8872f0fd085764fb; no voters:
I20260619 11:10:10.214897 14985 raft_consensus.cc:2781] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:10.215147 14985 raft_consensus.cc:686] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 LEADER]: Becoming Leader. State: Replica: 19795b4f8ef54fd6bcb9bb0c35d0ab7b, State: Running, Role: LEADER
I20260619 11:10:10.215306 14985 consensus_queue.cc:237] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } }
I20260619 11:10:10.215811 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.215570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:44685)
0619 11:10:10.215691 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.215747 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 11:10:10.216905 14242 catalog_manager.cc:5495] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67). New cstate: current_term: 1 leader_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19795b4f8ef54fd6bcb9bb0c35d0ab7b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42965 } health_report { overall_health: HEALTHY } } }
I20260619 11:10:10.228560 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.225415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.7.90.67:42965)
0619 11:10:10.225579 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:10.225583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:10.225601 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:10:10.225659 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:10.225661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:10.225700 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:10.225761 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:10.226033 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.226524 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.227606 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:10.227797 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:10.227864 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:10.228147 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:10.228207 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:10.228261 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:10.228376 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 11:10:10.228420 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:10.230472 14701 raft_consensus.cc:1261] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Refusing update from remote peer 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:10.230489 14564 raft_consensus.cc:1261] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Refusing update from remote peer 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:10.230904 14985 consensus_queue.cc:1035] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d40286feeb1440b8872f0fd085764fb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:10.231045 14988 consensus_queue.cc:1035] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89ad2e93eac4ebe884db870d7509399" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:10.232393 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.232095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:54033)
0619 11:10:10.232258 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.232317 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 11:10:10.233485 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.232832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:48641)
0619 11:10:10.233363 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.233413 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464}
I20260619 11:10:10.234208 14991 mvcc.cc:205] Tried to move back new op lower bound from 7298528912300359680 to 7298528912243048448. Current Snapshot: MvccSnapshot[applied={T|T < 7298528912300359680}]
I20260619 11:10:10.234962 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.234712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:50781)
0619 11:10:10.234846 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.234897 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 11:10:10.236742 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.236176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.66:51307)
0619 11:10:10.236631 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.236683 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389}
I20260619 11:10:10.237030 14992 mvcc.cc:205] Tried to move back new op lower bound from 7298528912300359680 to 7298528912243048448. Current Snapshot: MvccSnapshot[applied={T|T < 7298528912300359680}]
I20260619 11:10:10.239081 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.238849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.67:47323)
0619 11:10:10.238971 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.239019 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 11:10:10.239503 14242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:10.239655 14242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:10.240593 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.240325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:43695 (local address 127.7.90.65:55425)
0619 11:10:10.240448 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.240493 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:10:10.243829 14242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d29c390c01b440c7a0539b339fa22d01: Sending DeleteTablet for 3 replicas of tablet 8e82f1fe95e94146ba94bd41acfff393
I20260619 11:10:10.244362 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:10 UTC) from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:35796
I20260619 11:10:10.244606 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:10 UTC) from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:33960
I20260619 11:10:10.244628 14681 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:10 UTC) from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:45326
I20260619 11:10:10.244894 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:10 UTC) from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:33960
I20260619 11:10:10.244994 15000 tablet_replica.cc:330] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: stopping tablet replica
I20260619 11:10:10.245117 15000 raft_consensus.cc:2229] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:10.245152 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e82f1fe95e94146ba94bd41acfff393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:10 UTC) from {username='oryx', principal='oryx/127.7.90.125@KRBTEST.COM'} at 127.0.0.1:35796
I20260619 11:10:10.245285 15000 raft_consensus.cc:2258] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:10.245378 15001 tablet_replica.cc:330] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399: stopping tablet replica
I20260619 11:10:10.245534 15001 raft_consensus.cc:2229] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:10.245654 15001 raft_consensus.cc:2258] T 8e82f1fe95e94146ba94bd41acfff393 P c89ad2e93eac4ebe884db870d7509399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:10.245666 15002 tablet_replica.cc:330] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: stopping tablet replica
I20260619 11:10:10.245771 15002 raft_consensus.cc:2229] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:10.245776 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 14393
I20260619 11:10:10.247052 15000 ts_tablet_manager.cc:1905] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:10.249047 15000 ts_tablet_manager.cc:1918] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:10.249106 15000 log.cc:1196] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/8e82f1fe95e94146ba94bd41acfff393
I20260619 11:10:10.249414 15000 ts_tablet_manager.cc:1939] T 8e82f1fe95e94146ba94bd41acfff393 P 19795b4f8ef54fd6bcb9bb0c35d0ab7b: Deleting consensus metadata
I20260619 11:10:10.249635 15002 raft_consensus.cc:2258] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:10.250168 14234 catalog_manager.cc:4841] TS 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965): tablet 8e82f1fe95e94146ba94bd41acfff393 (table test-table [id=b17f7de2a100468285e35ce2d234b3dc]) successfully deleted
W20260619 11:10:10.250545 14234 catalog_manager.cc:4808] TS 19795b4f8ef54fd6bcb9bb0c35d0ab7b (127.7.90.67:42965): delete failed for tablet 8e82f1fe95e94146ba94bd41acfff393 because the tablet was not found. No further retry: Not found: Tablet not found: 8e82f1fe95e94146ba94bd41acfff393
I20260619 11:10:10.250541 15002 ts_tablet_manager.cc:1905] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:10.253700 15002 ts_tablet_manager.cc:1918] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:10.253784 15002 log.cc:1196] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/8e82f1fe95e94146ba94bd41acfff393
I20260619 11:10:10.253996 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 14618
I20260619 11:10:10.254148 15002 ts_tablet_manager.cc:1939] T 8e82f1fe95e94146ba94bd41acfff393 P 7d40286feeb1440b8872f0fd085764fb: Deleting consensus metadata
I20260619 11:10:10.254163 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.253889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43085 (local address 127.0.0.1:33966)
0619 11:10:10.253994 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.254071 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 11:10:10.255065 14234 catalog_manager.cc:4584] TS c89ad2e93eac4ebe884db870d7509399 (127.7.90.65:43085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e82f1fe95e94146ba94bd41acfff393: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43085: connect: Connection refused (error 111)
I20260619 11:10:10.260020 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 14754
I20260619 11:10:10.260983 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:10.260206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45477 (local address 127.0.0.1:45332)
0619 11:10:10.260820 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:10.260884 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:45477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513}
I20260619 11:10:10.266794 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 14110
I20260619 11:10:10.272544 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 14212
2026-06-19T11:10:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5425 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:10 dist-test-slave-f1t5 krb5kdc[15005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:10 dist-test-slave-f1t5 krb5kdc[15005](info): set up 2 sockets
Jun 19 11:10:10 dist-test-slave-f1t5 krb5kdc[15005](info): commencing operation
krb5kdc: starting...
W20260619 11:10:11.318934 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:11 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab.
Entry for principal oryx/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab.
I20260619 11:10:11.444132 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40219
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab
--principal=oryx/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:11.555135 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:11.555387 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:11.555426 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:11.559274 15018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:11.559321 15018 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:10:11.559384 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:11.559428 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:11.559463 15018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:11.559479 15018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:11.564659 15018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.126.keytab
--principal=oryx/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:11.565650 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:11.566622 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:11.575567 15023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:11.575577 15024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:11.575804 15018 server_base.cc:1034] running on GCE node
W20260619 11:10:11.575567 15026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:11.576627 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:11.578050 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:11.579259 15018 hybrid_clock.cc:648] HybridClock initialized: now 1781867411579241 us; error 31 us; skew 500 ppm
Jun 19 11:10:11 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:11.581964 15018 init.cc:377] Logged in from keytab as oryx/127.7.90.126@KRBTEST.COM (short username oryx)
I20260619 11:10:11.582883 15018 webserver.cc:460] Webserver started at http://127.7.90.126:45721/ using document root <none> and password file <none>
I20260619 11:10:11.583454 15018 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:11.583515 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:11.583734 15018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:11.585853 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "a24997703e76413c93f0c29cba120f19"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "344799dc32ccd017a6fe6d6ab9dcac9f"
server_key_iv: "6999caf3e274e4afef529c82ca3b485d"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.586434 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "a24997703e76413c93f0c29cba120f19"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "344799dc32ccd017a6fe6d6ab9dcac9f"
server_key_iv: "6999caf3e274e4afef529c82ca3b485d"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.590140 15018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:11.592516 15033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:11.593779 15018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:11.593911 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "a24997703e76413c93f0c29cba120f19"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "344799dc32ccd017a6fe6d6ab9dcac9f"
server_key_iv: "6999caf3e274e4afef529c82ca3b485d"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.594027 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:11.615523 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:11.618330 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:11.618543 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:11.626631 15018 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40219
I20260619 11:10:11.626643 15095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40219 every 8 connection(s)
I20260619 11:10:11.627516 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:11.630396 15096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:11.630388 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15018
I20260619 11:10:11.630522 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:11.630790 7529 external_mini_cluster.cc:1428] Setting key 1e6db3f618e6fa3d8cd4474093f686b5
I20260619 11:10:11.636953 15096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Bootstrap starting.
Jun 19 11:10:11 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:11.639288 15096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:11.640044 15096 log.cc:824] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:11.642181 15096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: No bootstrap required, opened a new log
I20260619 11:10:11.644887 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:11.632027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.7.90.126:40219)
0619 11:10:11.632381 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:11.632394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:11.632435 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:10:11.633040 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:11.633055 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:11.633377 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:11.633808 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:11.634578 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.635547 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.636437 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.636902 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.637097 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:11.639682 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:11.639687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:11.639700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:11.639724 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:11.642158 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.642667 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.642672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.642678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.642789 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.643080 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.643083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.643084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.643446 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:11.643642 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:11.643923 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 11:10:11.644206 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260619 11:10:11.645129 15096 raft_consensus.cc:348] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } }
I20260619 11:10:11.645387 15096 raft_consensus.cc:374] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:11.645479 15096 raft_consensus.cc:729] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24997703e76413c93f0c29cba120f19, State: Initialized, Role: FOLLOWER
I20260619 11:10:11.646021 15096 consensus_queue.cc:260] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } }
I20260619 11:10:11.646190 15096 raft_consensus.cc:388] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:11.646253 15096 raft_consensus.cc:482] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:11.646342 15096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:11.647369 15096 raft_consensus.cc:504] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } }
I20260619 11:10:11.647642 15096 leader_election.cc:302] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a24997703e76413c93f0c29cba120f19; no voters:
I20260619 11:10:11.647881 15096 leader_election.cc:288] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:11.648129 15101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:11.648315 15101 raft_consensus.cc:686] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [term 1 LEADER]: Becoming Leader. State: Replica: a24997703e76413c93f0c29cba120f19, State: Running, Role: LEADER
I20260619 11:10:11.648597 15101 consensus_queue.cc:237] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } }
I20260619 11:10:11.649031 15096 sys_catalog.cc:564] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:11.650137 15103 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a24997703e76413c93f0c29cba120f19. Latest consensus state: current_term: 1 leader_uuid: "a24997703e76413c93f0c29cba120f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } } }
I20260619 11:10:11.650264 15103 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:11.650599 15102 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a24997703e76413c93f0c29cba120f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24997703e76413c93f0c29cba120f19" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40219 } } }
I20260619 11:10:11.650686 15102 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:11.650866 15110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:11.653929 15110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:11.659149 15110 catalog_manager.cc:1347] Generated new cluster ID: 3ef2b1755598448bb52935752a797e2a
I20260619 11:10:11.659211 15110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:11.682629 15110 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:11.683955 15110 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:11.688803 15110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Generated new TSK 0
I20260619 11:10:11.689636 15110 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
I20260619 11:10:11.724215 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40219
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:11.835825 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:11.836086 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:11.836127 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:11.840178 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:11.840229 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:11.840265 15122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:10:11.840333 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:11.845295 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:11.846405 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:11.847374 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:11.858291 15128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:11.858321 15130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:11.858464 15122 server_base.cc:1034] running on GCE node
W20260619 11:10:11.858333 15127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:11.859038 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:11.859706 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:11.860885 15122 hybrid_clock.cc:648] HybridClock initialized: now 1781867411860866 us; error 41 us; skew 500 ppm
Jun 19 11:10:11 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:11.863669 15122 init.cc:377] Logged in from keytab as oryx/127.7.90.65@KRBTEST.COM (short username oryx)
I20260619 11:10:11.864455 15122 webserver.cc:460] Webserver started at http://127.7.90.65:41699/ using document root <none> and password file <none>
I20260619 11:10:11.865036 15122 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:11.865096 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:11.865257 15122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:11.867301 15122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "9b9f17eecf7c81ce5573e456b628ec23"
server_key_iv: "21156fe6b90836e5a28a519892d1e43b"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.867790 15122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "9b9f17eecf7c81ce5573e456b628ec23"
server_key_iv: "21156fe6b90836e5a28a519892d1e43b"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.871416 15122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:10:11.873624 15137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:11.874800 15122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:11.874888 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
format_stamp: "Formatted at 2026-06-19 11:10:11 on dist-test-slave-f1t5"
server_key: "9b9f17eecf7c81ce5573e456b628ec23"
server_key_iv: "21156fe6b90836e5a28a519892d1e43b"
server_key_version: "encryptionkey@0"
I20260619 11:10:11.874967 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:11.895727 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:11.899094 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:11.899294 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:11.900758 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:11.900805 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:11.900843 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:11.900889 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:11.910758 15122 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:46017
I20260619 11:10:11.910816 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:46017 every 8 connection(s)
Jun 19 11:10:11 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:11.911731 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:11.915161 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:11.901784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:52687 (local address 127.7.90.126:40219)
0619 11:10:11.901936 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:11.901941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:11.902805 (+ 864us) server_negotiation.cc:407] Connection header received
0619 11:10:11.904137 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:11.904142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:11.904211 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:11.904317 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:11.905939 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.906791 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.908362 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.908762 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.908889 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:11.911755 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:11.911761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:11.911766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:11.911795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:11.913330 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.913913 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.913915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.913916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.913993 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.914282 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.914284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.914285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.914445 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:11.914516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:11.914950 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 11:10:11.915046 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:10:11.915915 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:11.902204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.65:52687)
0619 11:10:11.902659 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:11.902707 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:11.903821 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:11.904507 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:11.904514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:11.905019 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:11.905757 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:11.905775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.906933 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.906937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:11.908231 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:11.908240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.908936 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.908942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:11.909152 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:11.909895 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:11.909912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:11.911583 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:11.913443 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:11.913447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:11.913457 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:11.913806 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:11.914073 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:11.914075 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:11.914076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:11.914196 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:11.914531 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:11.914535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:11.914740 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 11:10:11.914891 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 11:10:11.915145 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
I20260619 11:10:11.917052 15253 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40219
I20260619 11:10:11.917184 15253 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:11.917703 15253 heartbeater.cc:511] Master 127.7.90.126:40219 requested a full tablet report, sending...
I20260619 11:10:11.919162 15044 ts_manager.cc:194] Registered new tserver with Master: 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017)
I20260619 11:10:11.920513 15044 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:52687
I20260619 11:10:11.920789 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15122
I20260619 11:10:11.920908 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:11.921190 7529 external_mini_cluster.cc:1428] Setting key b1b53dc4e556abe47f59ce7c9c02c609
I20260619 11:10:11.926435 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:11.919145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.65:49751)
0619 11:10:11.919440 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:11.919455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:11.919543 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:11.919754 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:11.919757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:11.919960 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:11.920066 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:11.920073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.920878 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.920880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:11.921726 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:11.921737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.922427 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.922430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:11.922522 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:11.922995 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:11.923008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:11.923316 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:11.924993 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:11.924997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:11.924999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:11.925187 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:11.925471 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:11.925476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:11.925478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:11.925578 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:11.925915 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:11.925926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:11.926016 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:10:11.926122 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:10:11.926253 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260619 11:10:11.926436 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:11.919213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:49751 (local address 127.7.90.126:40219)
0619 11:10:11.919408 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:11.919411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:11.919524 (+ 113us) server_negotiation.cc:407] Connection header received
0619 11:10:11.919618 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:11.919620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:11.919654 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:11.919717 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:11.920167 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.920741 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.921846 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:11.922186 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:11.922332 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:11.923472 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:11.923476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:11.923478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:11.923505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:11.924861 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.925288 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.925292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.925294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.925366 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:11.925663 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:11.925666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:11.925668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:11.925819 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:11.925924 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:11.926180 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 11:10:11.926281 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
WARNING: no policy specified for oryx/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
I20260619 11:10:11.948895 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40219
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:12.057078 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:12.057299 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:12.057338 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:12.061223 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:12.061293 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:12.061331 15262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:10:12.061362 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:12.066545 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:12.067449 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:12.068321 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:12.077565 15268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:12.077653 15270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:12.077713 15262 server_base.cc:1034] running on GCE node
W20260619 11:10:12.077636 15267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:12.078233 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:12.078912 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:12.080075 15262 hybrid_clock.cc:648] HybridClock initialized: now 1781867412080041 us; error 45 us; skew 500 ppm
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:12.082969 15262 init.cc:377] Logged in from keytab as oryx/127.7.90.66@KRBTEST.COM (short username oryx)
I20260619 11:10:12.083804 15262 webserver.cc:460] Webserver started at http://127.7.90.66:33467/ using document root <none> and password file <none>
I20260619 11:10:12.084355 15262 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:12.084417 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:12.084648 15262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:12.086635 15262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "1d77313d426849bda492767814a5fcb2"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "952f1c379e57f2f76290ef5fe738fc70"
server_key_iv: "ec0c08c37fb10e519202119952b1ca57"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.087188 15262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "1d77313d426849bda492767814a5fcb2"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "952f1c379e57f2f76290ef5fe738fc70"
server_key_iv: "ec0c08c37fb10e519202119952b1ca57"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.090778 15262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:10:12.093091 15277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:12.094458 15262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:12.094621 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "1d77313d426849bda492767814a5fcb2"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "952f1c379e57f2f76290ef5fe738fc70"
server_key_iv: "ec0c08c37fb10e519202119952b1ca57"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.094731 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:12.110119 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:12.113478 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:12.114327 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:12.115938 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:12.115993 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:12.116070 15262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:12.116111 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:12.127899 15262 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:33155
I20260619 11:10:12.128098 15392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:33155 every 8 connection(s)
I20260619 11:10:12.128824 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:12.130769 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.116688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:41625 (local address 127.7.90.126:40219)
0619 11:10:12.116810 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.116816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.117986 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 11:10:12.119182 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.119186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.119247 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.119304 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.121210 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.122120 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.123807 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.124053 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.124178 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.127139 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:12.127142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:12.127144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:12.127167 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:12.128644 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.129241 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.129245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.129246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.129332 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.129753 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.129757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.129759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.130000 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:12.130090 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.130532 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.130646 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:10:12.131603 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.117271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.66:41625)
0619 11:10:12.117831 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.117870 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.118932 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.119775 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.119784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.120286 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:12.120996 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.121018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.122263 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.122270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.123696 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.123708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.124219 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.124223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.124414 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.125037 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:12.125052 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:12.126960 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:12.128762 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.128767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.128781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.129111 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.129463 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.129467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.129468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.129658 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.130111 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:12.130116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:12.130313 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.130453 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.130699 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":176,"threads_started":1}
I20260619 11:10:12.132925 15393 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40219
I20260619 11:10:12.133083 15393 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:12.133683 15393 heartbeater.cc:511] Master 127.7.90.126:40219 requested a full tablet report, sending...
I20260619 11:10:12.134791 15044 ts_manager.cc:194] Registered new tserver with Master: 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:12.135464 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15262
I20260619 11:10:12.135517 15044 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:41625
I20260619 11:10:12.135610 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:12.135993 7529 external_mini_cluster.cc:1428] Setting key bf05361db47dd8dd48bac575cd12d65a
I20260619 11:10:12.142709 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.135191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:54063 (local address 127.7.90.126:40219)
0619 11:10:12.135310 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.135315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.135579 (+ 264us) server_negotiation.cc:407] Connection header received
0619 11:10:12.135689 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.135692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.135749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.135839 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.136318 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.136930 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.137935 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.138164 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.138285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.139542 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:12.139545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:12.139547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:12.139568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:12.140879 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.141419 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.141422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.141423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.141523 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.141881 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.141883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.141885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.142078 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:12.142172 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.142501 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.142597 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:12.142930 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.135115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.66:54063)
0619 11:10:12.135468 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.135484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.135606 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.135855 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.135859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.136095 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:12.136197 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.136204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.137075 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.137079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.137788 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.137797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.138284 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.138288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.138383 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.138914 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:12.138935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:12.139374 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:12.140995 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.141002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.141007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.141307 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.141659 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.141665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.141669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.141782 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.142201 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:12.142208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:12.142325 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.142467 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.142641 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
I20260619 11:10:12.164533 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40219
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:12.275784 15402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:12.276124 15402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:12.276211 15402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:12.280792 15402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:12.280895 15402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:12.280977 15402 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:10:12.281059 15402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:12.287068 15402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.90.126:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:12.288444 15402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:12.289779 15402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:12.299721 15408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:12.299721 15407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:12.299926 15402 server_base.cc:1034] running on GCE node
W20260619 11:10:12.299839 15410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:12.300496 15402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:12.301175 15402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:12.302378 15402 hybrid_clock.cc:648] HybridClock initialized: now 1781867412302344 us; error 57 us; skew 500 ppm
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:12.305148 15402 init.cc:377] Logged in from keytab as oryx/127.7.90.67@KRBTEST.COM (short username oryx)
I20260619 11:10:12.306128 15402 webserver.cc:460] Webserver started at http://127.7.90.67:40269/ using document root <none> and password file <none>
I20260619 11:10:12.306654 15402 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:12.306713 15402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:12.306874 15402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:12.308894 15402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "edc5a63cf390470a81489170d9d3fcfc"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "cad6a72c3ac5f6cad36aa60f0e68e4c8"
server_key_iv: "89ed3c6849c16a0a8d6d669fae9c3338"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.309409 15402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "edc5a63cf390470a81489170d9d3fcfc"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "cad6a72c3ac5f6cad36aa60f0e68e4c8"
server_key_iv: "89ed3c6849c16a0a8d6d669fae9c3338"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.313194 15402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:12.315717 15417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:12.317039 15402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:12.317138 15402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "edc5a63cf390470a81489170d9d3fcfc"
format_stamp: "Formatted at 2026-06-19 11:10:12 on dist-test-slave-f1t5"
server_key: "cad6a72c3ac5f6cad36aa60f0e68e4c8"
server_key_iv: "89ed3c6849c16a0a8d6d669fae9c3338"
server_key_version: "encryptionkey@0"
I20260619 11:10:12.317221 15402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:12.332646 15402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:12.335585 15402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:12.335727 15402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:12.337059 15402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:12.337109 15402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:12.337147 15402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:12.337162 15402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:12.348060 15402 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44265
I20260619 11:10:12.348140 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44265 every 8 connection(s)
I20260619 11:10:12.349148 15402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:12.350457 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15402
I20260619 11:10:12.350636 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:12.350991 7529 external_mini_cluster.cc:1428] Setting key e0fc8d0610efdce0f9408c252442cee2
I20260619 11:10:12.352134 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.337833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:56373 (local address 127.7.90.126:40219)
0619 11:10:12.337980 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.337985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.338897 (+ 912us) server_negotiation.cc:407] Connection header received
0619 11:10:12.340203 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.340206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.340273 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.340336 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.342214 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.342830 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.344767 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.345139 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.345228 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.347901 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:12.347904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:12.347906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:12.347931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:12.349809 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.350416 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.350420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.350423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.350519 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.350906 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.350910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.350911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.351169 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:12.351278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.351855 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.351973 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:10:12.353057 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.338240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.67:56373)
0619 11:10:12.338713 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.338772 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.339905 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.340780 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.340793 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.341286 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:12.342030 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.342052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.342939 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.342944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.344593 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.344607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.345308 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.345311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.345581 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.346254 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:12.346270 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:12.347682 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:12.349957 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.349963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.349976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.350298 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.350678 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.350683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.350687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.350832 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.351324 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:12.351333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:12.351612 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.351806 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.352117 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260619 11:10:12.356547 15533 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40219
I20260619 11:10:12.356688 15533 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:12.357129 15533 heartbeater.cc:511] Master 127.7.90.126:40219 requested a full tablet report, sending...
I20260619 11:10:12.358196 15050 ts_manager.cc:194] Registered new tserver with Master: edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265)
I20260619 11:10:12.358978 15050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:56373
I20260619 11:10:12.364874 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.356712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.7.90.67:52599)
0619 11:10:12.357784 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.357801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.357899 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.358140 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.358142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.358348 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:12.358447 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.358452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.359395 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.359397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.360347 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.360357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.360775 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.360779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.360866 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.361331 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:12.361341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:12.361730 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:12.363185 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.363187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.363189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.363441 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.363765 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:12.363768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:12.363769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:12.363861 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:12.364286 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:12.364289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:12.364393 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.364536 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.364682 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":90,"threads_started":1}
I20260619 11:10:12.364897 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.356783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:52599 (local address 127.7.90.126:40219)
0619 11:10:12.356902 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.356906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.357880 (+ 974us) server_negotiation.cc:407] Connection header received
0619 11:10:12.357996 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.357999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.358046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.358135 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.358552 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.359264 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.360462 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.360668 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.360746 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.361844 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:12.361847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:12.361849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:12.361870 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:12.363093 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.363526 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.363528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.363529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.363645 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.363965 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.363970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.363972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.364186 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:12.364272 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.364632 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.364750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:10:12.365092 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:10:12.388199 15099 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:12.382002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.7.90.126:40219)
0619 11:10:12.382131 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.382135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.382146 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:10:12.382184 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.382187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.382224 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.382297 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.382970 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.383472 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.384431 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.384631 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.384699 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.387970 (+ 3271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43496: BlockingRecv error: recv got EOF from 127.0.0.1:43496 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:10:12 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:12.397583 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.389747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.7.90.126:40219)
0619 11:10:12.389871 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.389875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.390063 (+ 188us) server_negotiation.cc:407] Connection header received
0619 11:10:12.390151 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.390153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.390190 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.390257 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:12.390774 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.391258 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.392111 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.392323 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.392407 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.394618 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:12.394621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:12.394623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:12.394644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:12.395906 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.396341 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.396344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.396345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.396435 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:12.396692 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:12.396695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:12.396696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:12.396925 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:12.397092 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.397291 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.397383 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:12.400653 15050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:12.403776 15050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:12.416963 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.411589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:46017 (local address 127.0.0.1:44548)
0619 11:10:12.411959 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.412012 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.412140 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.412733 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.412739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.412766 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:12.412925 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.412946 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.414373 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.414377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.415495 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.415504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.416463 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.416466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.416570 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.416606 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.416715 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.416824 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260619 11:10:12.417429 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.411731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.7.90.65:46017)
0619 11:10:12.412131 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.412146 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.412172 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:10:12.412336 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.412346 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.412557 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.412726 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:12.413041 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.414259 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.415615 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.416128 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.416250 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.416405 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.416761 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.416870 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":81,"threads_started":1}
I20260619 11:10:12.417622 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.411815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:33155 (local address 127.0.0.1:50838)
0619 11:10:12.412312 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.412324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.412395 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.413713 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.413716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.413734 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:12.413838 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.413847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.415101 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.415104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.416094 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.416104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.417245 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.417248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.417330 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.417351 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.417415 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.417477 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":128,"threads_started":1}
I20260619 11:10:12.417692 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.412363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.7.90.66:33155)
0619 11:10:12.413227 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.413240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.413268 (+ 28us) server_negotiation.cc:407] Connection header received
0619 11:10:12.413341 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.413349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.413530 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.413736 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:12.413963 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.414995 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.416293 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.417132 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.417247 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.417393 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.417442 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.417523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":112,"threads_started":1}
I20260619 11:10:12.418499 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.412274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44265 (local address 127.0.0.1:60432)
0619 11:10:12.412930 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.413229 (+ 299us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.413324 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.413872 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.413878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.413898 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:12.413987 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.413993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.415004 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.415009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.416788 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.416797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.418125 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.418128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.418213 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.418229 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.418310 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.418374 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20260619 11:10:12.418777 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.412650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.7.90.67:44265)
0619 11:10:12.413034 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.413045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.413319 (+ 274us) server_negotiation.cc:407] Connection header received
0619 11:10:12.413428 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.413434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.413668 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.413811 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:12.414094 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.414891 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.416913 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.417762 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.417866 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.418023 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.418337 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.418421 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":92,"threads_started":1}
I20260619 11:10:12.420523 15327 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8e9bc1312747919c9819238bf71e20 (DEFAULT_TABLE table=test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.420521 15172 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8e9bc1312747919c9819238bf71e20 (DEFAULT_TABLE table=test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.421623 15327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8e9bc1312747919c9819238bf71e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.421622 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8e9bc1312747919c9819238bf71e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.423959 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8e9bc1312747919c9819238bf71e20 (DEFAULT_TABLE table=test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.424769 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8e9bc1312747919c9819238bf71e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.428978 15561 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Bootstrap starting.
I20260619 11:10:12.430334 15562 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Bootstrap starting.
I20260619 11:10:12.431289 15561 tablet_bootstrap.cc:654] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.431984 15563 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Bootstrap starting.
I20260619 11:10:12.432086 15562 tablet_bootstrap.cc:654] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.432291 15561 log.cc:824] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:12.432983 15562 log.cc:824] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:12.434227 15563 tablet_bootstrap.cc:654] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.434881 15563 log.cc:824] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:12.435112 15561 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: No bootstrap required, opened a new log
I20260619 11:10:12.435320 15561 ts_tablet_manager.cc:1397] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:10:12.435345 15562 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: No bootstrap required, opened a new log
I20260619 11:10:12.435519 15562 ts_tablet_manager.cc:1397] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:10:12.436862 15563 tablet_bootstrap.cc:492] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: No bootstrap required, opened a new log
I20260619 11:10:12.437139 15563 ts_tablet_manager.cc:1397] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:10:12.439785 15561 raft_consensus.cc:348] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.440089 15561 raft_consensus.cc:374] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.439976 15562 raft_consensus.cc:348] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.440150 15561 raft_consensus.cc:729] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.440201 15562 raft_consensus.cc:374] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.440261 15562 raft_consensus.cc:729] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d77313d426849bda492767814a5fcb2, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.440479 15563 raft_consensus.cc:348] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.440649 15562 consensus_queue.cc:260] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.440649 15561 consensus_queue.cc:260] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.440762 15563 raft_consensus.cc:374] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.440834 15563 raft_consensus.cc:729] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.441280 15563 consensus_queue.cc:260] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.441640 15253 heartbeater.cc:503] Master 127.7.90.126:40219 was elected leader, sending a full tablet report...
I20260619 11:10:12.441690 15561 ts_tablet_manager.cc:1428] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 11:10:12.441689 15562 ts_tablet_manager.cc:1428] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 11:10:12.442247 15563 ts_tablet_manager.cc:1428] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260619 11:10:12.442551 15533 heartbeater.cc:503] Master 127.7.90.126:40219 was elected leader, sending a full tablet report...
I20260619 11:10:12.442803 15393 heartbeater.cc:503] Master 127.7.90.126:40219 was elected leader, sending a full tablet report...
W20260619 11:10:12.600222 15534 tablet.cc:2369] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:10:12.630034 15394 tablet.cc:2369] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:10:12.663453 15254 tablet.cc:2369] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:12.758165 15569 raft_consensus.cc:482] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:12.758407 15569 raft_consensus.cc:504] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.759558 15569 leader_election.cc:288] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:12.763811 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.759870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:46017 (local address 127.7.90.67:41735)
0619 11:10:12.760102 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.760120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.760218 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.760528 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.760530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.760543 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:12.760794 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.760804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.761644 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.761646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.762789 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.762799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.763507 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.763510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.763578 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.763598 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.763655 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.763707 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3}
I20260619 11:10:12.763912 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.759931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:41735 (local address 127.7.90.65:46017)
0619 11:10:12.760169 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.760172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.760186 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:10:12.760325 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.760328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.760400 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.760502 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:12.760916 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.761506 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.762909 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.763393 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.763494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.763556 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.763702 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.763768 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260619 11:10:12.764348 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.759870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:33155 (local address 127.7.90.67:36809)
0619 11:10:12.760288 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:12.760301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:12.760383 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:12.760696 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:12.760698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:12.760710 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:12.761226 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.761233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.762056 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.762059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.763105 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:12.763113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.763945 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.763951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:12.764069 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.764095 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:10:12.764162 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:10:12.764220 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":36,"thread_start_us":115,"threads_started":1}
I20260619 11:10:12.764389 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.759931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:36809 (local address 127.7.90.66:33155)
0619 11:10:12.760080 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.760083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.760393 (+ 310us) server_negotiation.cc:407] Connection header received
0619 11:10:12.760449 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.760456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.760509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.760582 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:12.761346 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.761945 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.763217 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.763785 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.763883 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.763939 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.764196 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.764276 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 11:10:12.764533 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8e9bc1312747919c9819238bf71e20" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" is_pre_election: true
I20260619 11:10:12.764856 15207 raft_consensus.cc:2454] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:12.764909 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8e9bc1312747919c9819238bf71e20" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2" is_pre_election: true
I20260619 11:10:12.765254 15347 raft_consensus.cc:2454] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:12.765395 15420 leader_election.cc:302] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:12.765702 15569 raft_consensus.cc:2781] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:12.765794 15569 raft_consensus.cc:482] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:12.765838 15569 raft_consensus.cc:3037] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:12.766846 15569 raft_consensus.cc:504] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.767194 15569 leader_election.cc:288] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Requested vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:12.767464 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8e9bc1312747919c9819238bf71e20" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
I20260619 11:10:12.767501 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8e9bc1312747919c9819238bf71e20" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2"
I20260619 11:10:12.767591 15207 raft_consensus.cc:3037] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:12.767644 15347 raft_consensus.cc:3037] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:12.768572 15347 raft_consensus.cc:2454] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:12.768589 15207 raft_consensus.cc:2454] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:12.768898 15418 leader_election.cc:302] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d77313d426849bda492767814a5fcb2, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:12.769098 15569 raft_consensus.cc:2781] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:12.769285 15569 raft_consensus.cc:686] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Becoming Leader. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Running, Role: LEADER
I20260619 11:10:12.769658 15569 consensus_queue.cc:237] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } }
I20260619 11:10:12.772822 15046 catalog_manager.cc:5495] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc reported cstate change: term changed from 0 to 1, leader changed from <none> to edc5a63cf390470a81489170d9d3fcfc (127.7.90.67). New cstate: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260619 11:10:12.866413 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:12.863057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.7.90.67:44265)
0619 11:10:12.863180 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:12.863184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:12.863197 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:12.863271 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:12.863273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:12.863327 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:12.863411 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:12.863626 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.864124 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.864940 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:12.865215 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:12.865312 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:12.865586 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:12.866020 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:12.866106 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:12.866189 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 11:10:12.866234 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:10:12.871762 15347 raft_consensus.cc:1261] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:12.871778 15207 raft_consensus.cc:1261] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:12.872514 15571 consensus_queue.cc:1035] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:12.872690 15569 consensus_queue.cc:1035] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:12.885010 15576 mvcc.cc:205] Tried to move back new op lower bound from 7298528923115982848 to 7298528922707132416. Current Snapshot: MvccSnapshot[applied={T|T < 7298528923115982848 or (T in {7298528923115982848})}]
I20260619 11:10:12.887768 15050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:12.887948 15050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:12.890352 15050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Sending DeleteTablet for 3 replicas of tablet 6a8e9bc1312747919c9819238bf71e20
I20260619 11:10:12.891201 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8e9bc1312747919c9819238bf71e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:12 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44548
I20260619 11:10:12.891198 15327 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8e9bc1312747919c9819238bf71e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:12 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 11:10:12.891728 15584 tablet_replica.cc:330] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: stopping tablet replica
I20260619 11:10:12.891824 15050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:10:12.891870 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8e9bc1312747919c9819238bf71e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:12 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:60432
I20260619 11:10:12.892194 15584 raft_consensus.cc:2229] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 11:10:12.892333 15050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:12.892340 15583 tablet_replica.cc:330] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: stopping tablet replica
I20260619 11:10:12.892599 15583 raft_consensus.cc:2229] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:12.892724 15584 raft_consensus.cc:2258] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:12.892874 15583 raft_consensus.cc:2258] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:12.893219 15585 tablet_replica.cc:330] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: stopping tablet replica
I20260619 11:10:12.893467 15585 raft_consensus.cc:2229] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:12.893989 15585 raft_consensus.cc:2258] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:12.894860 15583 ts_tablet_manager.cc:1905] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:12.895200 15585 ts_tablet_manager.cc:1905] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:12.897487 15583 ts_tablet_manager.cc:1918] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:12.897605 15583 log.cc:1196] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/6a8e9bc1312747919c9819238bf71e20
I20260619 11:10:12.897925 15583 ts_tablet_manager.cc:1939] T 6a8e9bc1312747919c9819238bf71e20 P 1d77313d426849bda492767814a5fcb2: Deleting consensus metadata
I20260619 11:10:12.898373 15172 tablet_service.cc:1467] Processing CreateTablet for tablet 2013f0ea9ab34618b87324900aaefdeb (DEFAULT_TABLE table=test-table [id=d98837e1b18a4e4a871f0c1376fbac95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.898577 15327 tablet_service.cc:1467] Processing CreateTablet for tablet 2013f0ea9ab34618b87324900aaefdeb (DEFAULT_TABLE table=test-table [id=d98837e1b18a4e4a871f0c1376fbac95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.898639 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013f0ea9ab34618b87324900aaefdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.899082 15584 ts_tablet_manager.cc:1905] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:12.899469 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 2013f0ea9ab34618b87324900aaefdeb (DEFAULT_TABLE table=test-table [id=d98837e1b18a4e4a871f0c1376fbac95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:12.899699 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013f0ea9ab34618b87324900aaefdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.901396 15584 ts_tablet_manager.cc:1918] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:12.901511 15584 log.cc:1196] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/6a8e9bc1312747919c9819238bf71e20
I20260619 11:10:12.901862 15584 ts_tablet_manager.cc:1939] T 6a8e9bc1312747919c9819238bf71e20 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting consensus metadata
I20260619 11:10:12.901922 15563 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Bootstrap starting.
I20260619 11:10:12.902019 15561 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Bootstrap starting.
I20260619 11:10:12.902858 15563 tablet_bootstrap.cc:654] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.902906 15561 tablet_bootstrap.cc:654] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.903947 15036 catalog_manager.cc:4841] TS 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017): tablet 6a8e9bc1312747919c9819238bf71e20 (table test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]) successfully deleted
I20260619 11:10:12.904028 15563 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: No bootstrap required, opened a new log
I20260619 11:10:12.904098 15563 ts_tablet_manager.cc:1397] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 11:10:12.904532 15563 raft_consensus.cc:348] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.904656 15563 raft_consensus.cc:374] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.904691 15563 raft_consensus.cc:729] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.904812 15563 consensus_queue.cc:260] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.905077 15563 ts_tablet_manager.cc:1428] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:10:12.905195 15327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013f0ea9ab34618b87324900aaefdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:12.905853 15034 catalog_manager.cc:4841] TS 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155): tablet 6a8e9bc1312747919c9819238bf71e20 (table test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]) successfully deleted
I20260619 11:10:12.907704 15561 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: No bootstrap required, opened a new log
I20260619 11:10:12.907807 15561 ts_tablet_manager.cc:1397] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 11:10:12.908387 15585 ts_tablet_manager.cc:1918] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:12.908349 15561 raft_consensus.cc:348] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.908455 15561 raft_consensus.cc:374] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.908486 15561 raft_consensus.cc:729] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.908496 15585 log.cc:1196] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/6a8e9bc1312747919c9819238bf71e20
I20260619 11:10:12.908592 15561 consensus_queue.cc:260] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.908813 15585 ts_tablet_manager.cc:1939] T 6a8e9bc1312747919c9819238bf71e20 P edc5a63cf390470a81489170d9d3fcfc: Deleting consensus metadata
I20260619 11:10:12.908984 15562 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Bootstrap starting.
I20260619 11:10:12.909883 15562 tablet_bootstrap.cc:654] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:12.909880 15036 catalog_manager.cc:4841] TS edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265): tablet 6a8e9bc1312747919c9819238bf71e20 (table test-table [id=84e73887b0ba4d3ab9c4c15695f5c445]) successfully deleted
I20260619 11:10:12.911314 15561 ts_tablet_manager.cc:1428] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:10:12.912027 15562 tablet_bootstrap.cc:492] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: No bootstrap required, opened a new log
I20260619 11:10:12.912096 15562 ts_tablet_manager.cc:1397] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:12.912470 15562 raft_consensus.cc:348] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.912561 15562 raft_consensus.cc:374] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:12.912595 15562 raft_consensus.cc:729] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d77313d426849bda492767814a5fcb2, State: Initialized, Role: FOLLOWER
I20260619 11:10:12.912711 15562 consensus_queue.cc:260] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:12.912940 15562 ts_tablet_manager.cc:1428] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:10:13.010370 15569 raft_consensus.cc:482] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:13.010499 15569 raft_consensus.cc:504] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.011006 15569 leader_election.cc:288] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.011476 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2013f0ea9ab34618b87324900aaefdeb" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2" is_pre_election: true
I20260619 11:10:13.011479 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2013f0ea9ab34618b87324900aaefdeb" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" is_pre_election: true
I20260619 11:10:13.011634 15207 raft_consensus.cc:2454] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.011637 15347 raft_consensus.cc:2454] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.011909 15420 leader_election.cc:302] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.012120 15569 raft_consensus.cc:2781] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:13.012200 15569 raft_consensus.cc:482] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:13.012254 15569 raft_consensus.cc:3037] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.013182 15569 raft_consensus.cc:504] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.013567 15569 leader_election.cc:288] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Requested vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.013895 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2013f0ea9ab34618b87324900aaefdeb" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
I20260619 11:10:13.013944 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2013f0ea9ab34618b87324900aaefdeb" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2"
I20260619 11:10:13.014036 15207 raft_consensus.cc:3037] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.014065 15347 raft_consensus.cc:3037] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.014925 15207 raft_consensus.cc:2454] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.015203 15420 leader_election.cc:302] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.015271 15347 raft_consensus.cc:2454] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.015393 15569 raft_consensus.cc:2781] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:13.015479 15569 raft_consensus.cc:686] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Becoming Leader. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Running, Role: LEADER
I20260619 11:10:13.015625 15569 consensus_queue.cc:237] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.017508 15050 catalog_manager.cc:5495] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc reported cstate change: term changed from 0 to 1, leader changed from <none> to edc5a63cf390470a81489170d9d3fcfc (127.7.90.67). New cstate: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } health_report { overall_health: HEALTHY } } }
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867411, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.126@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:13.059762 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.051230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40219 (local address 127.0.0.1:43504)
0619 11:10:13.051607 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.051619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:13.051727 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:13.051999 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:13.052002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:13.052388 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:13.052483 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.052488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.053281 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.053284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.054022 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.054041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.054472 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.054474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.054536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.055185 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:13.055197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:13.056580 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:13.058345 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:13.058352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:13.058357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:13.058581 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:13.058819 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:13.058821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:13.058822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:13.058897 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:13.059277 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:13.059283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:13.059398 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 11:10:13.059510 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:10:13.059621 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
I20260619 11:10:13.059911 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.051257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.7.90.126:40219)
0619 11:10:13.051507 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.051514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.051727 (+ 213us) server_negotiation.cc:407] Connection header received
0619 11:10:13.051795 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.051797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.051841 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.051928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:13.052621 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.053160 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.054145 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.054365 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.054452 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.056728 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:13.056731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:13.056733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:13.056753 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:13.058245 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:13.058665 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:13.058667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:13.058668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:13.058742 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:13.058991 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:13.058993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:13.058994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:13.059152 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:13.059226 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.059598 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.059697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260619 11:10:13.063627 15050 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:43504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:10:13.064376 15050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:13.068974 15327 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3fb8c0762094d52afe4f6fb6ffae7e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:10:13.069001 15172 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3fb8c0762094d52afe4f6fb6ffae7e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:10:13.069001 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3fb8c0762094d52afe4f6fb6ffae7e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:10:13.069343 15327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.069355 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.069355 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8fd7e2f4ec454da6e255ce0c5fd15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.071993 15563 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: Bootstrap starting.
I20260619 11:10:13.071982 15562 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: Bootstrap starting.
I20260619 11:10:13.072887 15563 tablet_bootstrap.cc:654] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.072887 15562 tablet_bootstrap.cc:654] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.073307 15561 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: Bootstrap starting.
I20260619 11:10:13.074363 15561 tablet_bootstrap.cc:654] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.074438 15563 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: No bootstrap required, opened a new log
I20260619 11:10:13.074528 15563 ts_tablet_manager.cc:1397] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:10:13.074680 15562 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: No bootstrap required, opened a new log
I20260619 11:10:13.074762 15562 ts_tablet_manager.cc:1397] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:13.074941 15563 raft_consensus.cc:348] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.075045 15563 raft_consensus.cc:374] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.075078 15563 raft_consensus.cc:729] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.075198 15563 consensus_queue.cc:260] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.075205 15562 raft_consensus.cc:348] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.075300 15562 raft_consensus.cc:374] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.075335 15562 raft_consensus.cc:729] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d77313d426849bda492767814a5fcb2, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.075448 15562 consensus_queue.cc:260] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.075487 15563 ts_tablet_manager.cc:1428] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.075688 15562 ts_tablet_manager.cc:1428] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.075644 15571 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.075775 15571 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.075851 15561 tablet_bootstrap.cc:492] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: No bootstrap required, opened a new log
I20260619 11:10:13.075933 15561 ts_tablet_manager.cc:1397] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:13.076207 15568 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.076341 15568 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.076390 15561 raft_consensus.cc:348] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.076498 15561 raft_consensus.cc:374] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.076525 15561 raft_consensus.cc:729] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.076642 15561 consensus_queue.cc:260] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.076906 15567 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.077078 15567 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.077306 15561 ts_tablet_manager.cc:1428] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.171695 15571 raft_consensus.cc:482] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:13.171834 15571 raft_consensus.cc:504] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.172345 15571 leader_election.cc:288] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.172778 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8fd7e2f4ec454da6e255ce0c5fd15e" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2" is_pre_election: true
I20260619 11:10:13.172808 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8fd7e2f4ec454da6e255ce0c5fd15e" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" is_pre_election: true
I20260619 11:10:13.172950 15347 raft_consensus.cc:2454] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.172991 15207 raft_consensus.cc:2454] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.173292 15418 leader_election.cc:302] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d77313d426849bda492767814a5fcb2, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.173477 15571 raft_consensus.cc:2781] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:13.173553 15571 raft_consensus.cc:482] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:13.173610 15571 raft_consensus.cc:3037] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.174657 15571 raft_consensus.cc:504] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.174983 15571 leader_election.cc:288] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Requested vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.175325 15207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8fd7e2f4ec454da6e255ce0c5fd15e" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
I20260619 11:10:13.175325 15347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8fd7e2f4ec454da6e255ce0c5fd15e" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2"
I20260619 11:10:13.175447 15347 raft_consensus.cc:3037] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.175447 15207 raft_consensus.cc:3037] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.176501 15207 raft_consensus.cc:2454] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.176502 15347 raft_consensus.cc:2454] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.176776 15418 leader_election.cc:302] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d77313d426849bda492767814a5fcb2, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.176970 15571 raft_consensus.cc:2781] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:13.177069 15571 raft_consensus.cc:686] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Becoming Leader. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Running, Role: LEADER
I20260619 11:10:13.177223 15571 consensus_queue.cc:237] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.177943 15569 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: TxnStatusTablet state changed. Reason: New leader edc5a63cf390470a81489170d9d3fcfc. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.178038 15569 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:10:13.178287 15601 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:10:13.178443 15601 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:10:13.179076 15050 catalog_manager.cc:5495] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc reported cstate change: term changed from 0 to 1, leader changed from <none> to edc5a63cf390470a81489170d9d3fcfc (127.7.90.67). New cstate: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } health_report { overall_health: HEALTHY } } }
I20260619 11:10:13.199699 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.196136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.7.90.67:44265)
0619 11:10:13.196264 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.196271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.196286 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:10:13.196524 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.196526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.196570 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.196659 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:13.196915 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.197543 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.198550 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.198802 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.198918 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.199239 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.199316 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.199404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.199503 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.199555 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:10:13.199712 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.196012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44265 (local address 127.0.0.1:60446)
0619 11:10:13.196215 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.196231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:13.196354 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:13.196681 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:13.196684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:13.196693 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:10:13.196805 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.196809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.197687 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.197692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.198415 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.198425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.198927 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.198931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.199037 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.199094 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.199409 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.199426 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:10:13.199491 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:10:13.199571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 11:10:13.202093 15207 raft_consensus.cc:1261] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.202294 15347 raft_consensus.cc:1261] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.202530 15571 consensus_queue.cc:1035] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.202749 15569 consensus_queue.cc:1035] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.203873 15567 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: TxnStatusTablet state changed. Reason: New leader edc5a63cf390470a81489170d9d3fcfc. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.203990 15567 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.204192 15568 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: TxnStatusTablet state changed. Reason: New leader edc5a63cf390470a81489170d9d3fcfc. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.204317 15568 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.205583 15569 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.205715 15569 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:10:13.205811 15574 mvcc.cc:205] Tried to move back new op lower bound from 7298528924471545856 to 7298528924375298048. Current Snapshot: MvccSnapshot[applied={T|T < 7298528924471545856}]
I20260619 11:10:13.207248 15568 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.207369 15568 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 1d77313d426849bda492767814a5fcb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.209857 15569 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.209985 15569 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P edc5a63cf390470a81489170d9d3fcfc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:10:13.210923 15567 tablet_replica.cc:439] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } } }
I20260619 11:10:13.211025 15567 tablet_replica.cc:442] T 3e8fd7e2f4ec454da6e255ce0c5fd15e P 3a8c097f3e2d4d9798d32239b6fc7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:10:13.221457 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.218008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:47213 (local address 127.7.90.67:44265)
0619 11:10:13.218155 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.218160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.218407 (+ 247us) server_negotiation.cc:407] Connection header received
0619 11:10:13.218571 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.218575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.218621 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.218702 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:13.218930 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.219430 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.220275 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.220575 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.220705 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.220961 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.221037 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.221117 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.221212 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.221312 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":76}
I20260619 11:10:13.221470 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.217940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44265 (local address 127.7.90.67:47213)
0619 11:10:13.218322 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.218334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:13.218462 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:13.218709 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:13.218711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:13.218718 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:10:13.218819 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.218824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.219552 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.219555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.220155 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.220162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.220685 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.220688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.220780 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.220824 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.221118 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.221143 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:10:13.221207 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:10:13.221327 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":25,"thread_start_us":127,"threads_started":1}
I20260619 11:10:13.227666 15347 raft_consensus.cc:1261] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.227999 15207 raft_consensus.cc:1261] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.228108 15571 consensus_queue.cc:1035] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.228322 15569 consensus_queue.cc:1035] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.263034 15050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.263175 15050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.265257 15050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Sending DeleteTablet for 3 replicas of tablet 2013f0ea9ab34618b87324900aaefdeb
I20260619 11:10:13.265794 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet 2013f0ea9ab34618b87324900aaefdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44548
I20260619 11:10:13.265841 15327 tablet_service.cc:1514] Processing DeleteTablet for tablet 2013f0ea9ab34618b87324900aaefdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 11:10:13.266023 15584 tablet_replica.cc:330] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: stopping tablet replica
I20260619 11:10:13.266026 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 2013f0ea9ab34618b87324900aaefdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:60432
I20260619 11:10:13.266155 15584 raft_consensus.cc:2229] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.266278 15584 raft_consensus.cc:2258] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.266472 15583 tablet_replica.cc:330] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: stopping tablet replica
I20260619 11:10:13.266607 15583 raft_consensus.cc:2229] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.266731 15583 raft_consensus.cc:2258] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.266880 15585 tablet_replica.cc:330] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: stopping tablet replica
I20260619 11:10:13.266983 15585 raft_consensus.cc:2229] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:13.267088 15584 ts_tablet_manager.cc:1905] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.267118 15585 raft_consensus.cc:2258] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.267794 15585 ts_tablet_manager.cc:1905] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.270470 15585 ts_tablet_manager.cc:1918] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:10:13.270550 15585 log.cc:1196] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/2013f0ea9ab34618b87324900aaefdeb
I20260619 11:10:13.270723 15584 ts_tablet_manager.cc:1918] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:10:13.270809 15585 ts_tablet_manager.cc:1939] T 2013f0ea9ab34618b87324900aaefdeb P edc5a63cf390470a81489170d9d3fcfc: Deleting consensus metadata
I20260619 11:10:13.270817 15584 log.cc:1196] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/2013f0ea9ab34618b87324900aaefdeb
I20260619 11:10:13.271132 15584 ts_tablet_manager.cc:1939] T 2013f0ea9ab34618b87324900aaefdeb P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting consensus metadata
I20260619 11:10:13.271476 15036 catalog_manager.cc:4841] TS edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265): tablet 2013f0ea9ab34618b87324900aaefdeb (table test-table [id=d98837e1b18a4e4a871f0c1376fbac95]) successfully deleted
I20260619 11:10:13.271863 15036 catalog_manager.cc:4841] TS 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017): tablet 2013f0ea9ab34618b87324900aaefdeb (table test-table [id=d98837e1b18a4e4a871f0c1376fbac95]) successfully deleted
I20260619 11:10:13.274104 15583 ts_tablet_manager.cc:1905] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.276230 15583 ts_tablet_manager.cc:1918] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:10:13.276340 15583 log.cc:1196] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/2013f0ea9ab34618b87324900aaefdeb
I20260619 11:10:13.276571 15583 ts_tablet_manager.cc:1939] T 2013f0ea9ab34618b87324900aaefdeb P 1d77313d426849bda492767814a5fcb2: Deleting consensus metadata
I20260619 11:10:13.277271 15034 catalog_manager.cc:4841] TS 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155): tablet 2013f0ea9ab34618b87324900aaefdeb (table test-table [id=d98837e1b18a4e4a871f0c1376fbac95]) successfully deleted
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:10:13.298331 15595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:13.291570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.7.90.126:40219)
0619 11:10:13.291694 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.291698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.291821 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:10:13.291977 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.291980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.292029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.292100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:13.292968 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.293478 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.294440 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.294699 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.294821 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.298200 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43506: BlockingRecv error: recv got EOF from 127.0.0.1:43506 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM
I20260619 11:10:13.309449 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.299964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.7.90.126:40219)
0619 11:10:13.300089 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.300093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.300169 (+ 76us) server_negotiation.cc:407] Connection header received
0619 11:10:13.300301 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.300303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.300340 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.300405 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:13.301128 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.301733 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.302791 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.303007 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.303080 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.305754 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:13.305763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:13.305766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:13.305791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:13.307505 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:13.308033 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:13.308039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:13.308041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:13.308140 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:13.308486 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:13.308489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:13.308491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:13.308699 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:13.308855 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.309126 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.309257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:10:13.312942 15050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:13.313390 15050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:13.318610 15172 tablet_service.cc:1467] Processing CreateTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 (DEFAULT_TABLE table=test-table [id=633c957c4442494c88c168726f8a4735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.318936 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22f7615d054ce7b50e656c1ab8a9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.319100 15327 tablet_service.cc:1467] Processing CreateTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 (DEFAULT_TABLE table=test-table [id=633c957c4442494c88c168726f8a4735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.319376 15327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22f7615d054ce7b50e656c1ab8a9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.320973 15561 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Bootstrap starting.
I20260619 11:10:13.321856 15561 tablet_bootstrap.cc:654] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.321815 15467 tablet_service.cc:1467] Processing CreateTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 (DEFAULT_TABLE table=test-table [id=633c957c4442494c88c168726f8a4735]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.322068 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22f7615d054ce7b50e656c1ab8a9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.323297 15561 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: No bootstrap required, opened a new log
I20260619 11:10:13.323372 15561 ts_tablet_manager.cc:1397] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:13.323721 15561 raft_consensus.cc:348] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.323799 15561 raft_consensus.cc:374] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.323819 15561 raft_consensus.cc:729] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.323956 15561 consensus_queue.cc:260] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.324247 15561 ts_tablet_manager.cc:1428] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.324394 15563 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Bootstrap starting.
I20260619 11:10:13.325268 15563 tablet_bootstrap.cc:654] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.326519 15562 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Bootstrap starting.
I20260619 11:10:13.326582 15563 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: No bootstrap required, opened a new log
I20260619 11:10:13.326648 15563 ts_tablet_manager.cc:1397] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:13.327062 15563 raft_consensus.cc:348] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.327181 15563 raft_consensus.cc:374] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.327219 15563 raft_consensus.cc:729] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.327337 15563 consensus_queue.cc:260] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.327445 15562 tablet_bootstrap.cc:654] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.327659 15563 ts_tablet_manager.cc:1428] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.328617 15562 tablet_bootstrap.cc:492] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: No bootstrap required, opened a new log
I20260619 11:10:13.328706 15562 ts_tablet_manager.cc:1397] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:13.329152 15562 raft_consensus.cc:348] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.329269 15562 raft_consensus.cc:374] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.329308 15562 raft_consensus.cc:729] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d77313d426849bda492767814a5fcb2, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.329427 15562 consensus_queue.cc:260] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.329717 15562 ts_tablet_manager.cc:1428] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.589896 15567 raft_consensus.cc:482] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:13.590037 15567 raft_consensus.cc:504] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.591158 15567 leader_election.cc:288] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155), edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265)
I20260619 11:10:13.595299 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.591577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:60873 (local address 127.7.90.67:44265)
0619 11:10:13.591721 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.591728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.592181 (+ 453us) server_negotiation.cc:407] Connection header received
0619 11:10:13.592278 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.592281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.592336 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.592432 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:13.592692 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.593194 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.594348 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.594742 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.594864 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.594940 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.595103 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.595170 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:10:13.595325 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.591491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44265 (local address 127.7.90.65:60873)
0619 11:10:13.592081 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.592090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:13.592187 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:13.592442 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:13.592445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:13.592459 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:13.592576 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.592582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.593313 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.593318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.594216 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.594225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.594870 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.594874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.594971 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.594997 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:10:13.595087 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:10:13.595156 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":50,"thread_start_us":60,"threads_started":1}
I20260619 11:10:13.595625 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.591491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:33155 (local address 127.7.90.65:45851)
0619 11:10:13.591834 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.591847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:13.591947 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:13.592238 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:13.592241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:13.592260 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:13.592380 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.592392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.593193 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.593196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.594473 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:13.594482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.595164 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.595167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:13.595258 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.595284 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:10:13.595383 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 11:10:13.595475 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260619 11:10:13.595899 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.591581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:45851 (local address 127.7.90.66:33155)
0619 11:10:13.591851 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.591863 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.591897 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:10:13.592052 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.592054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.592103 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.592225 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:13.592498 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.593042 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.594607 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.595022 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.595155 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.595213 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.595699 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.595788 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260619 11:10:13.595840 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22f7615d054ce7b50e656c1ab8a9c2" candidate_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5a63cf390470a81489170d9d3fcfc" is_pre_election: true
I20260619 11:10:13.596112 15487 raft_consensus.cc:2454] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8c097f3e2d4d9798d32239b6fc7b63 in term 0.
I20260619 11:10:13.596532 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22f7615d054ce7b50e656c1ab8a9c2" candidate_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2" is_pre_election: true
I20260619 11:10:13.596572 15140 leader_election.cc:302] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.596692 15346 raft_consensus.cc:2454] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8c097f3e2d4d9798d32239b6fc7b63 in term 0.
I20260619 11:10:13.596890 15567 raft_consensus.cc:2781] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:13.596956 15567 raft_consensus.cc:482] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:13.597024 15567 raft_consensus.cc:3037] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.598289 15567 raft_consensus.cc:504] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.598702 15567 leader_election.cc:288] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [CANDIDATE]: Term 1 election: Requested vote from peers 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155), edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265)
I20260619 11:10:13.599131 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22f7615d054ce7b50e656c1ab8a9c2" candidate_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2"
I20260619 11:10:13.599150 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22f7615d054ce7b50e656c1ab8a9c2" candidate_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5a63cf390470a81489170d9d3fcfc"
I20260619 11:10:13.599269 15346 raft_consensus.cc:3037] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.599274 15487 raft_consensus.cc:3037] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.600222 15346 raft_consensus.cc:2454] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8c097f3e2d4d9798d32239b6fc7b63 in term 1.
I20260619 11:10:13.600374 15487 raft_consensus.cc:2454] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8c097f3e2d4d9798d32239b6fc7b63 in term 1.
I20260619 11:10:13.600559 15138 leader_election.cc:302] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d77313d426849bda492767814a5fcb2, 3a8c097f3e2d4d9798d32239b6fc7b63; no voters:
I20260619 11:10:13.600744 15567 raft_consensus.cc:2781] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:13.600909 15567 raft_consensus.cc:686] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Running, Role: LEADER
I20260619 11:10:13.601202 15567 consensus_queue.cc:237] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.603554 15044 catalog_manager.cc:5495] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65). New cstate: current_term: 1 leader_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20260619 11:10:13.620777 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.616751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.7.90.65:46017)
0619 11:10:13.617094 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.617101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.617112 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:10:13.617218 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.617221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.617274 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.617398 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:13.617680 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.618278 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.619247 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.619491 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.619603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.619880 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.620348 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.620492 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.620537 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.620585 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":151,"threads_started":1}
I20260619 11:10:13.623427 15346 raft_consensus.cc:1261] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Refusing update from remote peer 3a8c097f3e2d4d9798d32239b6fc7b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.623595 15487 raft_consensus.cc:1261] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Refusing update from remote peer 3a8c097f3e2d4d9798d32239b6fc7b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.623941 15567 consensus_queue.cc:1035] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.624136 15631 consensus_queue.cc:1035] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.629403 15575 mvcc.cc:205] Tried to move back new op lower bound from 7298528926196428800 to 7298528926113333248. Current Snapshot: MvccSnapshot[applied={T|T < 7298528926196428800 or (T in {7298528926196428800})}]
I20260619 11:10:13.635942 15044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.636089 15044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.638098 15044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Sending DeleteTablet for 3 replicas of tablet ee22f7615d054ce7b50e656c1ab8a9c2
I20260619 11:10:13.638664 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44548
I20260619 11:10:13.638816 15584 tablet_replica.cc:330] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: stopping tablet replica
I20260619 11:10:13.638819 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:60432
I20260619 11:10:13.638912 15584 raft_consensus.cc:2229] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:13.638978 15585 tablet_replica.cc:330] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: stopping tablet replica
I20260619 11:10:13.639084 15585 raft_consensus.cc:2229] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.639187 15584 raft_consensus.cc:2258] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.639206 15585 raft_consensus.cc:2258] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.639669 15050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:10:13.639840 15327 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22f7615d054ce7b50e656c1ab8a9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 11:10:13.640007 15583 tablet_replica.cc:330] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: stopping tablet replica
W20260619 11:10:13.640005 15050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:13.640105 15583 raft_consensus.cc:2229] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.640218 15583 raft_consensus.cc:2258] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.640861 15585 ts_tablet_manager.cc:1905] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.641963 15584 ts_tablet_manager.cc:1905] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.643344 15585 ts_tablet_manager.cc:1918] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:13.643438 15585 log.cc:1196] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/ee22f7615d054ce7b50e656c1ab8a9c2
I20260619 11:10:13.643693 15585 ts_tablet_manager.cc:1939] T ee22f7615d054ce7b50e656c1ab8a9c2 P edc5a63cf390470a81489170d9d3fcfc: Deleting consensus metadata
I20260619 11:10:13.643940 15584 ts_tablet_manager.cc:1918] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:13.644035 15584 log.cc:1196] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/ee22f7615d054ce7b50e656c1ab8a9c2
I20260619 11:10:13.644338 15584 ts_tablet_manager.cc:1939] T ee22f7615d054ce7b50e656c1ab8a9c2 P 3a8c097f3e2d4d9798d32239b6fc7b63: Deleting consensus metadata
I20260619 11:10:13.644394 15036 catalog_manager.cc:4841] TS edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265): tablet ee22f7615d054ce7b50e656c1ab8a9c2 (table test-table [id=633c957c4442494c88c168726f8a4735]) successfully deleted
I20260619 11:10:13.644951 15036 catalog_manager.cc:4841] TS 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017): tablet ee22f7615d054ce7b50e656c1ab8a9c2 (table test-table [id=633c957c4442494c88c168726f8a4735]) successfully deleted
I20260619 11:10:13.645032 15583 ts_tablet_manager.cc:1905] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.647281 15583 ts_tablet_manager.cc:1918] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:13.647343 15583 log.cc:1196] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/ee22f7615d054ce7b50e656c1ab8a9c2
I20260619 11:10:13.647593 15583 ts_tablet_manager.cc:1939] T ee22f7615d054ce7b50e656c1ab8a9c2 P 1d77313d426849bda492767814a5fcb2: Deleting consensus metadata
I20260619 11:10:13.648212 15034 catalog_manager.cc:4841] TS 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155): tablet ee22f7615d054ce7b50e656c1ab8a9c2 (table test-table [id=633c957c4442494c88c168726f8a4735]) successfully deleted
I20260619 11:10:13.649384 15172 tablet_service.cc:1467] Processing CreateTablet for tablet b480058a18164ad59f461f26c030a107 (DEFAULT_TABLE table=test-table [id=254c08016cf241a2b1d0ab8a1126136f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.649441 15467 tablet_service.cc:1467] Processing CreateTablet for tablet b480058a18164ad59f461f26c030a107 (DEFAULT_TABLE table=test-table [id=254c08016cf241a2b1d0ab8a1126136f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.649699 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b480058a18164ad59f461f26c030a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.649736 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b480058a18164ad59f461f26c030a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.651597 15561 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: Bootstrap starting.
I20260619 11:10:13.651889 15563 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Bootstrap starting.
I20260619 11:10:13.652633 15561 tablet_bootstrap.cc:654] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.652685 15563 tablet_bootstrap.cc:654] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.653623 15327 tablet_service.cc:1467] Processing CreateTablet for tablet b480058a18164ad59f461f26c030a107 (DEFAULT_TABLE table=test-table [id=254c08016cf241a2b1d0ab8a1126136f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:13.653889 15327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b480058a18164ad59f461f26c030a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:13.654008 15561 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: No bootstrap required, opened a new log
I20260619 11:10:13.654079 15561 ts_tablet_manager.cc:1397] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:13.654167 15563 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: No bootstrap required, opened a new log
I20260619 11:10:13.654235 15563 ts_tablet_manager.cc:1397] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:13.654542 15561 raft_consensus.cc:348] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.654567 15563 raft_consensus.cc:348] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.654654 15563 raft_consensus.cc:374] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.654654 15561 raft_consensus.cc:374] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.654686 15563 raft_consensus.cc:729] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.654686 15561 raft_consensus.cc:729] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8c097f3e2d4d9798d32239b6fc7b63, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.654801 15563 consensus_queue.cc:260] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.654801 15561 consensus_queue.cc:260] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.655057 15563 ts_tablet_manager.cc:1428] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.655054 15561 ts_tablet_manager.cc:1428] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:10:13.656059 15562 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Bootstrap starting.
I20260619 11:10:13.656935 15562 tablet_bootstrap.cc:654] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:13.658222 15562 tablet_bootstrap.cc:492] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: No bootstrap required, opened a new log
I20260619 11:10:13.658303 15562 ts_tablet_manager.cc:1397] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:13.658689 15562 raft_consensus.cc:348] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.658815 15562 raft_consensus.cc:374] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:13.658855 15562 raft_consensus.cc:729] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d77313d426849bda492767814a5fcb2, State: Initialized, Role: FOLLOWER
I20260619 11:10:13.658994 15562 consensus_queue.cc:260] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.659271 15562 ts_tablet_manager.cc:1428] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:10:13.788882 15609 raft_consensus.cc:482] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:13.789070 15609 raft_consensus.cc:504] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.789580 15609 leader_election.cc:288] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.790097 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b480058a18164ad59f461f26c030a107" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" is_pre_election: true
I20260619 11:10:13.790132 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b480058a18164ad59f461f26c030a107" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2" is_pre_election: true
I20260619 11:10:13.790275 15346 raft_consensus.cc:2454] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.790283 15206 raft_consensus.cc:2454] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 0.
I20260619 11:10:13.790669 15420 leader_election.cc:302] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.790882 15609 raft_consensus.cc:2781] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:13.790948 15609 raft_consensus.cc:482] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:13.790982 15609 raft_consensus.cc:3037] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.792197 15609 raft_consensus.cc:504] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.792549 15609 leader_election.cc:288] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Requested vote from peers 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017), 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155)
I20260619 11:10:13.792924 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b480058a18164ad59f461f26c030a107" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63"
I20260619 11:10:13.792959 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b480058a18164ad59f461f26c030a107" candidate_uuid: "edc5a63cf390470a81489170d9d3fcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d77313d426849bda492767814a5fcb2"
I20260619 11:10:13.793077 15346 raft_consensus.cc:3037] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.793107 15206 raft_consensus.cc:3037] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:13.794201 15206 raft_consensus.cc:2454] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.794236 15346 raft_consensus.cc:2454] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edc5a63cf390470a81489170d9d3fcfc in term 1.
I20260619 11:10:13.794557 15420 leader_election.cc:302] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8c097f3e2d4d9798d32239b6fc7b63, edc5a63cf390470a81489170d9d3fcfc; no voters:
I20260619 11:10:13.794759 15609 raft_consensus.cc:2781] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:13.794970 15609 raft_consensus.cc:686] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Becoming Leader. State: Replica: edc5a63cf390470a81489170d9d3fcfc, State: Running, Role: LEADER
I20260619 11:10:13.795135 15609 consensus_queue.cc:237] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } }
I20260619 11:10:13.796975 15046 catalog_manager.cc:5495] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc reported cstate change: term changed from 0 to 1, leader changed from <none> to edc5a63cf390470a81489170d9d3fcfc (127.7.90.67). New cstate: current_term: 1 leader_uuid: "edc5a63cf390470a81489170d9d3fcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5a63cf390470a81489170d9d3fcfc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44265 } health_report { overall_health: HEALTHY } } }
I20260619 11:10:13.821008 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.817633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.7.90.67:44265)
0619 11:10:13.817749 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:13.817755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:13.817808 (+ 53us) server_negotiation.cc:407] Connection header received
0619 11:10:13.818088 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:13.818092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:13.818140 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:13.818234 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:13.818485 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.819132 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.819989 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:13.820219 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:13.820332 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:13.820581 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:13.820658 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:13.820741 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:13.820836 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 11:10:13.820879 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:10:13.826642 15346 raft_consensus.cc:1261] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.826994 15206 raft_consensus.cc:1261] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Refusing update from remote peer edc5a63cf390470a81489170d9d3fcfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:13.827039 15641 consensus_queue.cc:1035] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d77313d426849bda492767814a5fcb2" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.827427 15643 consensus_queue.cc:1035] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8c097f3e2d4d9798d32239b6fc7b63" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:13.859305 15046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.859452 15046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:13.861266 15046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24997703e76413c93f0c29cba120f19: Sending DeleteTablet for 3 replicas of tablet b480058a18164ad59f461f26c030a107
I20260619 11:10:13.861891 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet b480058a18164ad59f461f26c030a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44548
I20260619 11:10:13.861903 15327 tablet_service.cc:1514] Processing DeleteTablet for tablet b480058a18164ad59f461f26c030a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:50838
I20260619 11:10:13.861903 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet b480058a18164ad59f461f26c030a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:13 UTC) from {username='oryx', principal='oryx/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:60432
I20260619 11:10:13.862104 15584 tablet_replica.cc:330] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63: stopping tablet replica
I20260619 11:10:13.862224 15584 raft_consensus.cc:2229] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.862362 15584 raft_consensus.cc:2258] T b480058a18164ad59f461f26c030a107 P 3a8c097f3e2d4d9798d32239b6fc7b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.862562 15583 tablet_replica.cc:330] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: stopping tablet replica
I20260619 11:10:13.862684 15583 raft_consensus.cc:2229] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:13.862795 15583 raft_consensus.cc:2258] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.862972 15585 tablet_replica.cc:330] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: stopping tablet replica
I20260619 11:10:13.863023 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15122
I20260619 11:10:13.863097 15585 raft_consensus.cc:2229] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:13.863294 15585 raft_consensus.cc:2258] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:13.864289 15585 ts_tablet_manager.cc:1905] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.865170 15583 ts_tablet_manager.cc:1905] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:13.866868 15585 ts_tablet_manager.cc:1918] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:10:13.866961 15585 log.cc:1196] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/b480058a18164ad59f461f26c030a107
I20260619 11:10:13.867204 15585 ts_tablet_manager.cc:1939] T b480058a18164ad59f461f26c030a107 P edc5a63cf390470a81489170d9d3fcfc: Deleting consensus metadata
I20260619 11:10:13.867862 15036 catalog_manager.cc:4841] TS edc5a63cf390470a81489170d9d3fcfc (127.7.90.67:44265): tablet b480058a18164ad59f461f26c030a107 (table test-table [id=254c08016cf241a2b1d0ab8a1126136f]) successfully deleted
I20260619 11:10:13.868191 15583 ts_tablet_manager.cc:1918] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:10:13.868275 15583 log.cc:1196] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/b480058a18164ad59f461f26c030a107
I20260619 11:10:13.868549 15583 ts_tablet_manager.cc:1939] T b480058a18164ad59f461f26c030a107 P 1d77313d426849bda492767814a5fcb2: Deleting consensus metadata
I20260619 11:10:13.869237 15034 catalog_manager.cc:4841] TS 1d77313d426849bda492767814a5fcb2 (127.7.90.66:33155): tablet b480058a18164ad59f461f26c030a107 (table test-table [id=254c08016cf241a2b1d0ab8a1126136f]) successfully deleted
W20260619 11:10:13.872910 15036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:46017 (error 108)
I20260619 11:10:13.872963 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15262
I20260619 11:10:13.874835 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:13.874334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:46017 (local address 127.0.0.1:44570)
0619 11:10:13.874612 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:13.874701 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
W20260619 11:10:13.875020 15036 catalog_manager.cc:4584] TS 3a8c097f3e2d4d9798d32239b6fc7b63 (127.7.90.65:46017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b480058a18164ad59f461f26c030a107: Network error: Client connection negotiation failed: client connection to 127.7.90.65:46017: connect: Connection refused (error 111)
I20260619 11:10:13.879556 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15402
I20260619 11:10:13.886826 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15018
2026-06-19T11:10:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3613 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15652](info): set up 2 sockets
Jun 19 11:10:13 dist-test-slave-f1t5 krb5kdc[15652](info): commencing operation
krb5kdc: starting...
W20260619 11:10:14.934249 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:15 dist-test-slave-f1t5 krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:10:15.056257 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43813
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:15.165573 15665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:15.165824 15665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:15.165906 15665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:15.169816 15665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:15.169869 15665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:15.169888 15665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:15.169952 15665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:15.169981 15665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:15.175196 15665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:15.176239 15665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:15.177227 15665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:15.185379 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:15.185387 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:15.185387 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:15.185599 15665 server_base.cc:1034] running on GCE node
I20260619 11:10:15.186249 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:15.187309 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:15.188501 15665 hybrid_clock.cc:648] HybridClock initialized: now 1781867415188474 us; error 52 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T11:10:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1315 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:15 dist-test-slave-f1t5 krb5kdc[15678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:15 dist-test-slave-f1t5 krb5kdc[15678](info): set up 2 sockets
Jun 19 11:10:15 dist-test-slave-f1t5 krb5kdc[15678](info): commencing operation
krb5kdc: starting...
W20260619 11:10:16.242138 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:10:16.360080 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45411
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:16.469413 15691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:16.469708 15691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:16.469748 15691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:16.473591 15691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:16.473660 15691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:16.473681 15691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:16.473697 15691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:16.473713 15691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:16.478874 15691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:45411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:16.479858 15691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:16.480819 15691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:16.489464 15696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:16.489454 15699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:16.489552 15691 server_base.cc:1034] running on GCE node
W20260619 11:10:16.489454 15697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:16.490252 15691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:16.491263 15691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:16.492452 15691 hybrid_clock.cc:648] HybridClock initialized: now 1781867416492424 us; error 42 us; skew 500 ppm
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:16.495121 15691 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:10:16.495985 15691 webserver.cc:460] Webserver started at http://127.7.90.126:45867/ using document root <none> and password file <none>
I20260619 11:10:16.496536 15691 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:16.496629 15691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:16.496857 15691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:16.498912 15691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "16ff562f806f4154a76d8459700a4907"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "7c2f9f9fae4eb15ad9cd429804afea88"
server_key_iv: "8f39f3895512b205f07f44fb60a9a456"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.499469 15691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "16ff562f806f4154a76d8459700a4907"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "7c2f9f9fae4eb15ad9cd429804afea88"
server_key_iv: "8f39f3895512b205f07f44fb60a9a456"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.503115 15691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:10:16.505353 15706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:16.506682 15691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:16.506842 15691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "16ff562f806f4154a76d8459700a4907"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "7c2f9f9fae4eb15ad9cd429804afea88"
server_key_iv: "8f39f3895512b205f07f44fb60a9a456"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.506964 15691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:16.521812 15691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:16.524979 15691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:16.525197 15691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:16.532483 15691 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:45411
I20260619 11:10:16.532502 15758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:45411 every 8 connection(s)
I20260619 11:10:16.533396 15691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:16.536036 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:16.536331 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15691
I20260619 11:10:16.536460 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:16.536754 7529 external_mini_cluster.cc:1428] Setting key 5605b5b584649b70f3e768b22e85c0a2
I20260619 11:10:16.542941 15759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Bootstrap starting.
I20260619 11:10:16.545289 15759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Neither blocks nor log segments found. Creating new log.
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:16.546195 15759 log.cc:824] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:16.548017 15759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: No bootstrap required, opened a new log
I20260619 11:10:16.551347 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:16.538131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.7.90.126:45411)
0619 11:10:16.538522 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:16.538540 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:16.538581 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:10:16.539306 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:16.539329 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:16.539756 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:16.540127 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:16.540758 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.542312 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.543199 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.543845 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.544037 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:16.546678 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:16.546684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:16.546701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:16.546733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:16.548538 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:16.549008 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:16.549012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:16.549016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:16.549125 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:16.549423 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:16.549425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:16.549426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:16.549848 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:16.550050 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:16.550375 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:10:16.550664 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
I20260619 11:10:16.551949 15759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:16.552196 15759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:16.552270 15759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ff562f806f4154a76d8459700a4907, State: Initialized, Role: FOLLOWER
I20260619 11:10:16.552726 15759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:16.552846 15759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:16.552927 15759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:16.553021 15759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:16.554039 15759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:16.554374 15759 leader_election.cc:302] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ff562f806f4154a76d8459700a4907; no voters:
I20260619 11:10:16.554666 15759 leader_election.cc:288] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:16.554787 15764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:16.554999 15764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 LEADER]: Becoming Leader. State: Replica: 16ff562f806f4154a76d8459700a4907, State: Running, Role: LEADER
I20260619 11:10:16.555328 15764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:16.556110 15759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:16.557067 15766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ff562f806f4154a76d8459700a4907. Latest consensus state: current_term: 1 leader_uuid: "16ff562f806f4154a76d8459700a4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } } }
I20260619 11:10:16.557199 15766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:16.557505 15765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ff562f806f4154a76d8459700a4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } } }
I20260619 11:10:16.557624 15765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:16.557658 15773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:16.560611 15773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:16.565960 15773 catalog_manager.cc:1347] Generated new cluster ID: 6b3ef419b3df402283145d149b4a854c
I20260619 11:10:16.566031 15773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:16.579525 15773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:16.580857 15773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:16.592082 15773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Generated new TSK 0
I20260619 11:10:16.592782 15773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:10:16.624339 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:16.742900 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:16.743173 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:16.743234 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:16.747164 15785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:16.747212 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:16.747267 15785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:16.752362 15785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:16.753288 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:16.754257 15785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:16.763773 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:16.763788 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:16.763788 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:16.763865 15785 server_base.cc:1034] running on GCE node
I20260619 11:10:16.764422 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:16.764978 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:16.766122 15785 hybrid_clock.cc:648] HybridClock initialized: now 1781867416766090 us; error 47 us; skew 500 ppm
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:16.768711 15785 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:10:16.769600 15785 webserver.cc:460] Webserver started at http://127.7.90.65:44423/ using document root <none> and password file <none>
I20260619 11:10:16.770195 15785 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:16.770282 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:16.770498 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:16.772477 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "d416918ef8e34ddd81666e26677a246c"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "b58623bc3811f420ce92478f70f774ca"
server_key_iv: "5b5ebe677b12ff4f9d5d4bcb7c5c7ba9"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.773108 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "d416918ef8e34ddd81666e26677a246c"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "b58623bc3811f420ce92478f70f774ca"
server_key_iv: "5b5ebe677b12ff4f9d5d4bcb7c5c7ba9"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.776926 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 11:10:16.779431 15800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:16.780794 15785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:16.780933 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "d416918ef8e34ddd81666e26677a246c"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "b58623bc3811f420ce92478f70f774ca"
server_key_iv: "5b5ebe677b12ff4f9d5d4bcb7c5c7ba9"
server_key_version: "encryptionkey@0"
I20260619 11:10:16.781056 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:16.805680 15785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:16.809198 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:16.809415 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:16.810177 15785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:16.811132 15785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:16.811209 15785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:16.811290 15785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:16.811339 15785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:16.820397 15785 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:44845
I20260619 11:10:16.820428 15913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:44845 every 8 connection(s)
I20260619 11:10:16.821355 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:16.831108 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15785
I20260619 11:10:16.831254 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:16.831641 7529 external_mini_cluster.cc:1428] Setting key 9fac0996123bde0ae4b86da55add5ee0
I20260619 11:10:16.835209 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:16.823240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:34571 (local address 127.7.90.126:45411)
0619 11:10:16.823398 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:16.823403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:16.824442 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 11:10:16.825255 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:16.825258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:16.825301 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:16.825372 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:16.826664 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.827204 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.828279 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.828491 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.828572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:16.830871 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:16.830875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:16.830878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:16.830907 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:16.832523 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:16.833195 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:16.833206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:16.833208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:16.833301 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:16.833966 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:16.833969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:16.833970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:16.834148 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:16.834230 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:16.834875 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 11:10:16.834973 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 11:10:16.836120 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:16.823649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:34571)
0619 11:10:16.824283 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:16.824324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:16.825053 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:16.825589 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:16.825603 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:16.825933 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:16.826506 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:16.826526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.827311 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.827314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:16.828179 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:16.828187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:16.828613 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:16.828616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:16.828749 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:16.829279 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:16.829295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:16.830698 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:16.832641 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:16.832646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:16.832658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:16.833026 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:16.833620 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:16.833623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:16.833625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:16.833824 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:16.834267 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:16.834277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:16.834618 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 11:10:16.834823 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 11:10:16.835179 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":108,"threads_started":1}
I20260619 11:10:16.837337 15914 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45411
I20260619 11:10:16.837642 15914 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:16.838268 15914 heartbeater.cc:511] Master 127.7.90.126:45411 requested a full tablet report, sending...
I20260619 11:10:16.839809 15723 ts_manager.cc:194] Registered new tserver with Master: d416918ef8e34ddd81666e26677a246c (127.7.90.65:44845)
I20260619 11:10:16.840904 15723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:34571
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:10:16.860250 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:16.969134 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:16.969414 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:16.969485 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:16.973448 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:16.973537 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:16.973682 15919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:16.978849 15919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:16.979928 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:16.980924 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:16.990424 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:16.990456 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:16.990599 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:16.990649 15919 server_base.cc:1034] running on GCE node
I20260619 11:10:16.991166 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:16.991775 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:16.992980 15919 hybrid_clock.cc:648] HybridClock initialized: now 1781867416992962 us; error 33 us; skew 500 ppm
Jun 19 11:10:16 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:16.995663 15919 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:10:16.996492 15919 webserver.cc:460] Webserver started at http://127.7.90.66:42185/ using document root <none> and password file <none>
I20260619 11:10:16.997054 15919 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:16.997145 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:16.997372 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:16.999434 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "ec5c03857f4f4b2287da6dd2bc2da9c9"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "551f4a08fad99ba22c3c049ab3a3194f"
server_key_iv: "ee2d045fdd872f2fa67ad44dab0dcb2c"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.000005 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "ec5c03857f4f4b2287da6dd2bc2da9c9"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "551f4a08fad99ba22c3c049ab3a3194f"
server_key_iv: "ee2d045fdd872f2fa67ad44dab0dcb2c"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.003808 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:10:17.007089 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.008255 15919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:17.008402 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "ec5c03857f4f4b2287da6dd2bc2da9c9"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "551f4a08fad99ba22c3c049ab3a3194f"
server_key_iv: "ee2d045fdd872f2fa67ad44dab0dcb2c"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.008531 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:17.026324 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:17.029167 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:17.029371 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:17.029944 15919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:17.030730 15919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:17.030802 15919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.030867 15919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:17.030917 15919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.039444 15919 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35865
I20260619 11:10:17.039515 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35865 every 8 connection(s)
I20260619 11:10:17.040349 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:17.046437 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 15919
I20260619 11:10:17.046554 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:17.046875 7529 external_mini_cluster.cc:1428] Setting key 7f356022d0f3b18806162eb099893365
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867416, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:17.054629 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.042056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:40837 (local address 127.7.90.126:45411)
0619 11:10:17.042181 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.042185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.043206 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 11:10:17.044073 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.044077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.044127 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.044207 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.045278 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.045888 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.047600 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.047819 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.047896 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.050813 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:17.050819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:17.050824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:17.050855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:17.052335 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:17.052999 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:17.053002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:17.053003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:17.053074 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:17.053470 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:17.053472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:17.053473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:17.053697 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:17.053775 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:17.054424 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 11:10:17.054519 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:17.055759 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.042452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:40837)
0619 11:10:17.043067 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.043105 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.043875 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.044361 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.044368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.044705 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.045140 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.045152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.046050 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.046054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.047436 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.047448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.047961 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.047967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.048133 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.048676 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.048696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.050623 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:17.052497 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:17.052506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:17.052522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:17.052884 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:17.053168 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:17.053174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:17.053179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:17.053371 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:17.053806 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:17.053815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:17.054159 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 11:10:17.054383 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 11:10:17.054719 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":110,"threads_started":1}
I20260619 11:10:17.056957 16048 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45411
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:10:17.057210 16048 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:17.057776 16048 heartbeater.cc:511] Master 127.7.90.126:45411 requested a full tablet report, sending...
I20260619 11:10:17.058776 15723 ts_manager.cc:194] Registered new tserver with Master: ec5c03857f4f4b2287da6dd2bc2da9c9 (127.7.90.66:35865)
I20260619 11:10:17.059454 15723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:40837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:10:17.074394 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:17.182489 16053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:17.182797 16053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:17.182868 16053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:17.187237 16053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:17.187320 16053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:17.187409 16053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:17.192634 16053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:17.193759 16053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:17.194756 16053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:17.204066 16058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.204054 16053 server_base.cc:1034] running on GCE node
W20260619 11:10:17.204121 16061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:17.204121 16059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.204811 16053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:17.205466 16053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:17.206678 16053 hybrid_clock.cc:648] HybridClock initialized: now 1781867417206664 us; error 25 us; skew 500 ppm
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867417, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:17.209232 16053 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:10:17.210166 16053 webserver.cc:460] Webserver started at http://127.7.90.67:41305/ using document root <none> and password file <none>
I20260619 11:10:17.210719 16053 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:17.210811 16053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:17.211025 16053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:17.213200 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "fa2e8698ba12410f9518b89c6805b016"
format_stamp: "Formatted at 2026-06-19 11:10:17 on dist-test-slave-f1t5"
server_key: "b7d48a74d56d4c6dd232427cfa7b7c0b"
server_key_iv: "e91273d42a4f088d3efdfb9b19ec1757"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.213810 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "fa2e8698ba12410f9518b89c6805b016"
format_stamp: "Formatted at 2026-06-19 11:10:17 on dist-test-slave-f1t5"
server_key: "b7d48a74d56d4c6dd232427cfa7b7c0b"
server_key_iv: "e91273d42a4f088d3efdfb9b19ec1757"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.217293 16053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:10:17.219511 16068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.220654 16053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:17.220770 16053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "fa2e8698ba12410f9518b89c6805b016"
format_stamp: "Formatted at 2026-06-19 11:10:17 on dist-test-slave-f1t5"
server_key: "b7d48a74d56d4c6dd232427cfa7b7c0b"
server_key_iv: "e91273d42a4f088d3efdfb9b19ec1757"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.220865 16053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:17.236743 16053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:17.239451 16053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:17.239655 16053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:17.240212 16053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:17.240965 16053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:17.241017 16053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.241086 16053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:17.241123 16053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.249538 16053 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:43139
I20260619 11:10:17.249617 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:43139 every 8 connection(s)
I20260619 11:10:17.250473 16053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867417, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:17.260278 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16053
I20260619 11:10:17.260423 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:17.260747 7529 external_mini_cluster.cc:1428] Setting key 9dfea05eff476647f8186856d0515621
I20260619 11:10:17.263358 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.252159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:54245 (local address 127.7.90.126:45411)
0619 11:10:17.252291 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.252295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.253312 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 11:10:17.254185 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.254188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.254229 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.254301 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.255329 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.255828 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.256912 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.257143 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.257263 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.259498 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:17.259501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:17.259504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:17.259526 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:17.261263 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:17.261817 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:17.261822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:17.261824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:17.261910 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:17.262255 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:17.262258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:17.262260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:17.262479 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:17.262594 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:17.263082 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 11:10:17.263195 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:10:17.264067 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.252550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:54245)
0619 11:10:17.253178 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.253215 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.254004 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.254449 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.254454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.254768 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.255208 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.255220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.255930 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.255933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.256748 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.256756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.257302 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.257306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.257462 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.258012 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.258027 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.259346 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:17.261365 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:17.261369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:17.261379 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:17.261717 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:17.262019 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:17.262021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:17.262023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:17.262147 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:17.262596 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:17.262603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:17.262833 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 11:10:17.263017 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:10:17.263314 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":100,"threads_started":1}
I20260619 11:10:17.265122 16182 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45411
I20260619 11:10:17.265349 16182 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:17.265867 16182 heartbeater.cc:511] Master 127.7.90.126:45411 requested a full tablet report, sending...
I20260619 11:10:17.266834 15723 ts_manager.cc:194] Registered new tserver with Master: fa2e8698ba12410f9518b89c6805b016 (127.7.90.67:43139)
I20260619 11:10:17.267323 15723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:54245
I20260619 11:10:17.274396 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
Entry for principal oryx/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab.
WARNING: no policy specified for oryx/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
Entry for principal oryx/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab.
WARNING: no policy specified for oryx/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
Entry for principal oryx/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab.
I20260619 11:10:17.347472 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15785
I20260619 11:10:17.352929 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15919
I20260619 11:10:17.358404 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16053
I20260619 11:10:17.363585 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 15691
I20260619 11:10:17.369395 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45411
--webserver_interface=127.7.90.126
--webserver_port=45867
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:17.476269 16191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:17.476590 16191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:17.476665 16191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:17.480636 16191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:17.480751 16191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:17.480787 16191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:17.480827 16191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:17.480856 16191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:17.486075 16191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:45411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:17.487166 16191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:17.488206 16191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:17.496726 16196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.496791 16191 server_base.cc:1034] running on GCE node
W20260619 11:10:17.496731 16199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:17.496870 16197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.497680 16191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:17.498781 16191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:17.499941 16191 hybrid_clock.cc:648] HybridClock initialized: now 1781867417499924 us; error 31 us; skew 500 ppm
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867417, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:17.502861 16191 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:10:17.503786 16191 webserver.cc:460] Webserver started at http://127.7.90.126:45867/ using document root <none> and password file <none>
I20260619 11:10:17.504396 16191 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:17.504505 16191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:17.508527 16191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:17.510951 16206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.512200 16191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:17.512346 16191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "16ff562f806f4154a76d8459700a4907"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "7c2f9f9fae4eb15ad9cd429804afea88"
server_key_iv: "8f39f3895512b205f07f44fb60a9a456"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.512943 16191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:17.533830 16191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:17.537098 16191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:17.537297 16191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:17.544463 16191 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:45411
I20260619 11:10:17.544489 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:45411 every 8 connection(s)
I20260619 11:10:17.545451 16191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:17.549499 16259 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:10:17.551092 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Bootstrap starting.
I20260619 11:10:17.555109 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16191
I20260619 11:10:17.555613 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:44845
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:45411
--webserver_port=44423
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:10:17.571992 16259 log.cc:824] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:17.580299 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:10:17.580704 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Bootstrap complete.
I20260619 11:10:17.586223 16259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:17.586714 16259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ff562f806f4154a76d8459700a4907, State: Initialized, Role: FOLLOWER
I20260619 11:10:17.587245 16259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:17.587386 16259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:17.587448 16259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:17.587550 16259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:10:17.590034 16259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:17.590426 16259 leader_election.cc:302] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ff562f806f4154a76d8459700a4907; no voters:
I20260619 11:10:17.590755 16259 leader_election.cc:288] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:10:17.590875 16263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:10:17.591111 16263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [term 2 LEADER]: Becoming Leader. State: Replica: 16ff562f806f4154a76d8459700a4907, State: Running, Role: LEADER
I20260619 11:10:17.591466 16263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } }
I20260619 11:10:17.592100 16259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:17.593026 16264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16ff562f806f4154a76d8459700a4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } } }
I20260619 11:10:17.593179 16264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:17.593441 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ff562f806f4154a76d8459700a4907. Latest consensus state: current_term: 2 leader_uuid: "16ff562f806f4154a76d8459700a4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ff562f806f4154a76d8459700a4907" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45411 } } }
I20260619 11:10:17.593560 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:17.597916 16278 catalog_manager.cc:1259] Loaded cluster ID: 6b3ef419b3df402283145d149b4a854c
I20260619 11:10:17.598017 16278 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: loading cluster ID for follower catalog manager: success
I20260619 11:10:17.600407 16278 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: acquiring CA information for follower catalog manager: success
I20260619 11:10:17.601969 16278 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:10:17.602131 16272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:17.602995 16272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:17.603410 16272 catalog_manager.cc:1259] Loaded cluster ID: 6b3ef419b3df402283145d149b4a854c
I20260619 11:10:17.603451 16272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:17.604012 16272 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:17.604519 16272 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16ff562f806f4154a76d8459700a4907: Loaded TSK: 0
I20260619 11:10:17.605237 16272 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:10:17.682627 16261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:17.682878 16261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:17.682922 16261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:17.686980 16261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:17.687041 16261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:17.687101 16261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:17.692871 16261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.65.keytab
--principal=oryx/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:44845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=44423
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:17.694149 16261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:17.695178 16261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:17.706204 16285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:17.706305 16288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:17.706249 16286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.706701 16261 server_base.cc:1034] running on GCE node
I20260619 11:10:17.707180 16261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:17.707854 16261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:17.709061 16261 hybrid_clock.cc:648] HybridClock initialized: now 1781867417709032 us; error 47 us; skew 500 ppm
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867417, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:17.711886 16261 init.cc:377] Logged in from keytab as oryx/127.7.90.65@KRBTEST.COM (short username oryx)
I20260619 11:10:17.712828 16261 webserver.cc:460] Webserver started at http://127.7.90.65:44423/ using document root <none> and password file <none>
I20260619 11:10:17.713384 16261 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:17.713462 16261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:17.717470 16261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:17.719632 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.720933 16261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:17.721089 16261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "d416918ef8e34ddd81666e26677a246c"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "b58623bc3811f420ce92478f70f774ca"
server_key_iv: "5b5ebe677b12ff4f9d5d4bcb7c5c7ba9"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.721676 16261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:17.741250 16261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:17.744150 16261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:17.744341 16261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:17.744904 16261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:17.745702 16261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:17.745767 16261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.745836 16261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:17.745874 16261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.754415 16261 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:44845
I20260619 11:10:17.754452 16408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:44845 every 8 connection(s)
I20260619 11:10:17.755342 16261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:17.763358 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16261
I20260619 11:10:17.763896 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:35865
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:45411
--webserver_port=42185
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.767357 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.757256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:43273)
0619 11:10:17.757934 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.757982 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.758717 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.759686 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.759692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.760018 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.760455 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.760468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.761700 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.761703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.762592 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.762601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.763283 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.763287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.763507 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.764069 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.764094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.766655 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":130,"threads_started":1}
W20260619 11:10:17.767800 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.757158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:43273 (local address 127.7.90.126:45411)
0619 11:10:17.757590 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.757605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.758095 (+ 490us) server_negotiation.cc:407] Connection header received
0619 11:10:17.759069 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.759080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.759355 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.759613 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.760590 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.761511 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.762691 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.763132 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.763242 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.766922 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:43273: BlockingRecv error: recv got EOF from 127.7.90.65:43273 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
W20260619 11:10:17.767877 16409 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45411 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.90.126:45411: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.773780 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.768271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:50909)
0619 11:10:17.768421 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.768437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.768532 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.768802 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.768804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.768956 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.769047 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.769052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.769984 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.769990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.770560 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.770569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.771034 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.771038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.771145 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.771654 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.771666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.773620 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 11:10:17.774118 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.768485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:50909 (local address 127.7.90.126:45411)
0619 11:10:17.768615 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.768619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.768631 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:17.768666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.768668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.768704 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.768788 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.769170 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.769828 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.770685 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.770922 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.771037 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.773751 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:50909: BlockingRecv error: recv got EOF from 127.7.90.65:50909 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.779523 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.774328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:46025)
0619 11:10:17.774439 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.774453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.774540 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.774988 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.774990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.775109 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.775176 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.775181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.776129 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.776132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.776624 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.776631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.777178 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.777181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.777251 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.777722 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.777734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.779436 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:10:17.779675 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.774599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:46025 (local address 127.7.90.126:45411)
0619 11:10:17.774737 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.774743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.774761 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:10:17.774800 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.774805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.774856 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.774971 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.775274 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.776023 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.776726 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.777073 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.777166 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.779577 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:46025: BlockingRecv error: recv got EOF from 127.7.90.65:46025 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:10:17.779801 16409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:10:17.878613 16413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:17.878909 16413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:17.878983 16413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:17.882994 16413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:17.883090 16413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:17.883184 16413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:17.888414 16413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.66.keytab
--principal=oryx/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:35865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=42185
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:17.889534 16413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:17.890580 16413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:17.900429 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.900462 16413 server_base.cc:1034] running on GCE node
W20260619 11:10:17.900369 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:17.900369 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:17.901010 16413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:17.901762 16413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:17.902925 16413 hybrid_clock.cc:648] HybridClock initialized: now 1781867417902912 us; error 23 us; skew 500 ppm
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867417, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:17.905731 16413 init.cc:377] Logged in from keytab as oryx/127.7.90.66@KRBTEST.COM (short username oryx)
I20260619 11:10:17.906613 16413 webserver.cc:460] Webserver started at http://127.7.90.66:42185/ using document root <none> and password file <none>
I20260619 11:10:17.907207 16413 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:17.907282 16413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:17.911095 16413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:10:17.912997 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.914249 16413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:17.914369 16413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "ec5c03857f4f4b2287da6dd2bc2da9c9"
format_stamp: "Formatted at 2026-06-19 11:10:16 on dist-test-slave-f1t5"
server_key: "551f4a08fad99ba22c3c049ab3a3194f"
server_key_iv: "ee2d045fdd872f2fa67ad44dab0dcb2c"
server_key_version: "encryptionkey@0"
I20260619 11:10:17.914892 16413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:17.927407 16413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:17.930641 16413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:17.930819 16413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:17.931375 16413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:17.932165 16413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:17.932209 16413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.932271 16413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:17.932303 16413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:17.940639 16413 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35865
I20260619 11:10:17.940716 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35865 every 8 connection(s)
I20260619 11:10:17.941692 16413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.950798 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16413
I20260619 11:10:17.951170 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:43139
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:45411
--webserver_port=41305
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:17.951960 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.943419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:34317 (local address 127.7.90.126:45411)
0619 11:10:17.943550 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.943566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.944587 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 11:10:17.945391 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.945394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.945439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.945502 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.946546 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.947077 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.948117 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.948314 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.948384 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.951792 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:34317: BlockingRecv error: recv got EOF from 127.7.90.66:34317 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:10:17.952411 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.943824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:34317)
0619 11:10:17.944437 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.944475 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.945217 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.945666 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.945672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.945999 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.946434 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.946445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.947190 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.947192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.948024 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.948031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.948421 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.948424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.948548 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.949019 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.949052 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.951619 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":115,"threads_started":1}
W20260619 11:10:17.953022 16542 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45411 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.90.126:45411: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.958211 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.953430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:35909)
0619 11:10:17.953553 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.953577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.953660 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.953882 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.953885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.954035 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.954116 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.954120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.954799 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.954805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.955343 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.955350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.955768 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.955772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.955861 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.956346 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.956356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.958077 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:10:17.958303 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.953546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:35909 (local address 127.7.90.126:45411)
0619 11:10:17.953663 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.953667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.953682 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:10:17.953747 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.953749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.953790 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.953869 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.954219 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.954700 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.955455 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.955666 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.955769 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.958192 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:35909: BlockingRecv error: recv got EOF from 127.7.90.66:35909 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:17 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:17.963393 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:17.958875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:48487)
0619 11:10:17.958979 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:17.958996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:17.959078 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:17.959290 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:17.959293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:17.959437 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:17.959499 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.959509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.960183 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.960190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.960696 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:17.960703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.961112 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.961115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:17.961191 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.961661 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:17.961671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:17.963273 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:10:17.963620 16542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:10:17.963753 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:17.958961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:48487 (local address 127.7.90.126:45411)
0619 11:10:17.959066 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:17.959070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:17.959083 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:17.959156 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:17.959159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:17.959200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:17.959277 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:17.959612 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.960087 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.960805 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:17.961009 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:17.961110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:17.963376 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:48487: BlockingRecv error: recv got EOF from 127.7.90.66:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 11:10:18.061058 16545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:18.061299 16545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:18.061353 16545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:18.065357 16545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:18.065426 16545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:18.065523 16545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:18.070894 16545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/krb5kdc/oryx_127.7.90.67.keytab
--principal=oryx/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:43139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=41305
--tserver_master_addrs=127.7.90.126:45411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:18.071978 16545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:18.073036 16545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:18.082705 16550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:18.082700 16553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:18.082768 16551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:18.082983 16545 server_base.cc:1034] running on GCE node
I20260619 11:10:18.083429 16545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:18.084091 16545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:18.085268 16545 hybrid_clock.cc:648] HybridClock initialized: now 1781867418085231 us; error 50 us; skew 500 ppm
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867418, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:18.087963 16545 init.cc:377] Logged in from keytab as oryx/127.7.90.67@KRBTEST.COM (short username oryx)
I20260619 11:10:18.088894 16545 webserver.cc:460] Webserver started at http://127.7.90.67:41305/ using document root <none> and password file <none>
I20260619 11:10:18.089416 16545 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:18.089486 16545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:18.093637 16545 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:10:18.095940 16560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:18.097424 16545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:10:18.097599 16545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "fa2e8698ba12410f9518b89c6805b016"
format_stamp: "Formatted at 2026-06-19 11:10:17 on dist-test-slave-f1t5"
server_key: "b7d48a74d56d4c6dd232427cfa7b7c0b"
server_key_iv: "e91273d42a4f088d3efdfb9b19ec1757"
server_key_version: "encryptionkey@0"
I20260619 11:10:18.098161 16545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:18.125277 16545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:18.128360 16545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:18.128515 16545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:18.129101 16545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:18.129930 16545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:18.129977 16545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:18.130017 16545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:18.130033 16545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:18.138432 16545 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:43139
I20260619 11:10:18.138502 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:43139 every 8 connection(s)
I20260619 11:10:18.139410 16545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:18.147888 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16545
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:10:18.150470 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:18.141173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:45835 (local address 127.7.90.126:45411)
0619 11:10:18.141313 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.141316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.142144 (+ 828us) server_negotiation.cc:407] Connection header received
0619 11:10:18.143421 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.143424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.143476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.143578 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.144694 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.145207 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.146375 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.146587 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.146688 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.150375 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:45835: BlockingRecv error: recv got EOF from 127.7.90.67:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:10:18.151185 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.141572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:45835)
0619 11:10:18.142006 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:18.142048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:18.143222 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:18.143760 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:18.143767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:18.144107 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:18.144549 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.144571 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.145325 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.145328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.146257 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.146266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.146721 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.146725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.146899 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.147448 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:18.147463 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:18.150281 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
W20260619 11:10:18.151685 16674 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45411 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.90.126:45411: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:18.157114 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.152118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:52819)
0619 11:10:18.152258 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:18.152271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:18.152337 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:18.152567 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:18.152569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:18.152688 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:18.152768 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.152772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.153453 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.153456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.154290 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.154299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.154711 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.154714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.154815 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.155148 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:18.155161 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:18.156995 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 11:10:18.157156 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:18.152193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:52819 (local address 127.7.90.126:45411)
0619 11:10:18.152290 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.152293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.152320 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:10:18.152420 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.152423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.152464 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.152545 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.152884 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.153359 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.154402 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.154612 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.154711 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.157076 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:52819: BlockingRecv error: recv got EOF from 127.7.90.67:52819 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:10:18.158900 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.148907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.7.90.126:45411)
0619 11:10:18.149189 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.149192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.149203 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:10:18.149232 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.149234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.149264 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.149337 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.150124 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.150881 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.151928 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.152250 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.152358 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.153854 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:18.153861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:18.153879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:18.153907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:18.156156 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:18.156584 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:18.156591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:18.156597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:18.156739 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:18.157333 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:18.157336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:18.157338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:18.157888 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:18.158092 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:18.158514 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 11:10:18.158753 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:18.162663 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.157696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:48397)
0619 11:10:18.157791 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:18.157805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:18.157872 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:18.158204 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:18.158207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:18.158363 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:18.158440 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.158448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.159405 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.159408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.159905 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.159912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.160335 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.160339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.160415 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.160858 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:18.160872 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:18.162582 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:10:18.162781 16411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:18.157730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:48397 (local address 127.7.90.126:45411)
0619 11:10:18.157836 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.157839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.157850 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:10:18.157958 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.157962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.158003 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.158071 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.158601 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.159311 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.160022 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.160235 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.160336 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.162685 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:48397: BlockingRecv error: recv got EOF from 127.7.90.67:48397 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 11:10:18.162899 16674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:18.785918 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.780442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:41375)
0619 11:10:18.780738 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:18.780751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:18.780853 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:18.781110 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:18.781112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:18.781281 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:18.781390 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.781395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.782195 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.782197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.782761 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.782768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.783197 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.783199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.783268 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.783815 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:18.783827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:18.785741 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
W20260619 11:10:18.786059 16680 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:18.780558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:41375 (local address 127.7.90.126:45411)
0619 11:10:18.780771 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.780775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.780817 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:10:18.780944 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.780947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.780990 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.781068 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.781511 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.782091 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.782861 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.783100 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.783181 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.785905 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:41375: BlockingRecv error: recv got EOF from 127.7.90.65:41375 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:18 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:18.970045 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:18.964314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:52501)
0619 11:10:18.964603 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:18.964617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:18.964753 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:18.965030 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:18.965033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:18.965236 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:18.965332 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.965339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.966177 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.966181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.966767 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:18.966776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.967266 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.967270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:18.967365 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.967962 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:18.967974 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:18.969863 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260619 11:10:18.970073 16680 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:18.964374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:52501 (local address 127.7.90.126:45411)
0619 11:10:18.964543 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:18.964550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:18.964690 (+ 140us) server_negotiation.cc:407] Connection header received
0619 11:10:18.964850 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:18.964855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:18.964907 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:18.965008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:18.965450 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.966053 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.966889 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:18.967139 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:18.967262 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:18.969974 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:52501: BlockingRecv error: recv got EOF from 127.7.90.66:52501 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:19.169333 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:19.163537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:37953)
0619 11:10:19.163815 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:19.163834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:19.163988 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:19.164256 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:19.164259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:19.164477 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:19.164567 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.164573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.165596 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.165601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.166437 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.166446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.166874 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.166876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.166933 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.167437 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:19.167451 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:19.169216 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
W20260619 11:10:19.169512 16680 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:19.163625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:37953 (local address 127.7.90.126:45411)
0619 11:10:19.163815 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:19.163821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:19.163927 (+ 106us) server_negotiation.cc:407] Connection header received
0619 11:10:19.164091 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:19.164094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:19.164148 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:19.164237 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:19.164686 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.165458 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.166560 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.166779 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.166864 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.169348 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:37953: BlockingRecv error: recv got EOF from 127.7.90.67:37953 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:19.791776 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:19.786697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:60785)
0619 11:10:19.786986 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:19.786999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:19.787099 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:19.787341 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:19.787343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:19.787515 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:19.787609 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.787613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.788307 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.788308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.788844 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.788851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.789242 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.789245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.789306 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.789860 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:19.789872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:19.791637 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
W20260619 11:10:19.791882 16684 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:19.786881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:60785 (local address 127.7.90.126:45411)
0619 11:10:19.787148 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:19.787151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:19.787163 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:10:19.787196 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:19.787198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:19.787236 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:19.787310 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:19.787701 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.788219 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.788941 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.789154 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.789225 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.791761 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:60785: BlockingRecv error: recv got EOF from 127.7.90.65:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":123,"threads_started":1}
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:19 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:19.975800 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:19.970838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:46159)
0619 11:10:19.971100 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:19.971112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:19.971208 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:19.971424 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:19.971426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:19.971603 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:19.971699 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.971704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.972393 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.972395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.972918 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:19.972924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.973324 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.973327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:19.973387 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.973948 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:19.973959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:19.975663 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
W20260619 11:10:19.975911 16684 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:19.970918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:46159 (local address 127.7.90.126:45411)
0619 11:10:19.971102 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:19.971105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:19.971178 (+ 73us) server_negotiation.cc:407] Connection header received
0619 11:10:19.971283 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:19.971285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:19.971327 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:19.971391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:19.971788 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.972312 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.973014 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:19.973234 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:19.973310 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:19.975783 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:46159: BlockingRecv error: recv got EOF from 127.7.90.66:46159 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:20.175568 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:20.170104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:33933)
0619 11:10:20.170357 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:20.170387 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:20.170479 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:20.170719 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:20.170721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:20.170884 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:20.170974 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.170979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.171669 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.171671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.172247 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.172254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.172659 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.172662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.172733 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.173230 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:20.173242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:20.175344 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260619 11:10:20.175624 16684 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:20.170238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:33933 (local address 127.7.90.126:45411)
0619 11:10:20.170353 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:20.170356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:20.170449 (+ 93us) server_negotiation.cc:407] Connection header received
0619 11:10:20.170565 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:20.170568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:20.170611 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:20.170685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:20.171078 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.171584 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.172334 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.172534 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.172613 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.175490 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:33933: BlockingRecv error: recv got EOF from 127.7.90.67:33933 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:20.801854 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:20.792466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:60863)
0619 11:10:20.792809 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:20.792830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:20.792930 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:20.793220 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:20.793223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:20.793430 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:20.793584 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.793591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.794339 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.794341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.794921 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.794929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.795515 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.795518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.795587 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.799773 (+ 4186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:20.799789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:20.801736 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
W20260619 11:10:20.802000 16688 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:20.792605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:60863 (local address 127.7.90.126:45411)
0619 11:10:20.792885 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:20.792889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:20.792902 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:20.793015 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:20.793017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:20.793085 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:20.793160 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:20.793692 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.794238 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.795017 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.795241 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.795329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.801869 (+ 6540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:60863: BlockingRecv error: recv got EOF from 127.7.90.65:60863 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:20 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:20.982437 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:20.976620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:53219)
0619 11:10:20.976945 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:20.976963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:20.977100 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:20.977352 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:20.977355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:20.977586 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:20.977678 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.977684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.978436 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.978440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.979000 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:20.979007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.979459 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.979463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:20.979551 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.980086 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:20.980098 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:20.982245 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260619 11:10:20.982466 16688 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:20.976743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:53219 (local address 127.7.90.126:45411)
0619 11:10:20.976923 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:20.976928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:20.977042 (+ 114us) server_negotiation.cc:407] Connection header received
0619 11:10:20.977196 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:20.977199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:20.977251 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:20.977334 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:20.977794 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.978326 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.979122 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:20.979355 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:20.979458 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:20.982367 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:53219: BlockingRecv error: recv got EOF from 127.7.90.66:53219 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:21.182014 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:21.176515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:33155)
0619 11:10:21.176794 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:21.176807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:21.176907 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:21.177209 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:21.177211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:21.177386 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:21.177478 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.177483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.178468 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.178472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.179037 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.179045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.179426 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.179429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.179487 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.179956 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:21.179971 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:21.181821 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
W20260619 11:10:21.182040 16688 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:21.176594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:33155 (local address 127.7.90.126:45411)
0619 11:10:21.176743 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:21.176747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:21.176864 (+ 117us) server_negotiation.cc:407] Connection header received
0619 11:10:21.177032 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:21.177034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:21.177080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:21.177147 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:21.177620 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.178362 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.179144 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.179342 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.179414 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.181940 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:33155: BlockingRecv error: recv got EOF from 127.7.90.67:33155 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:21.807648 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:21.802594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:58607)
0619 11:10:21.802836 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:21.802850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:21.802970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:21.803273 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:21.803275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:21.803445 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:21.803547 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.803552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.804272 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.804274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.804787 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.804793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.805180 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.805183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.805238 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.805820 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:21.805838 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:21.807526 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
W20260619 11:10:21.807811 16692 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:21.802644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:58607 (local address 127.7.90.126:45411)
0619 11:10:21.802885 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:21.802889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:21.802919 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:10:21.803110 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:21.803113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:21.803159 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:21.803244 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:21.803641 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.804178 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.804879 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.805094 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.805166 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.807652 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:58607: BlockingRecv error: recv got EOF from 127.7.90.65:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:21 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:21.988601 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:21.983358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:47401)
0619 11:10:21.983703 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:21.983715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:21.983808 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:21.984081 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:21.984083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:21.984346 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:21.984484 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.984490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.985186 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.985188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.985750 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:21.985757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.986148 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.986150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:21.986208 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.986744 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:21.986756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:21.988488 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":80,"threads_started":1}
W20260619 11:10:21.988777 16692 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:21.983398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:47401 (local address 127.7.90.126:45411)
0619 11:10:21.983565 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:21.983568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:21.983769 (+ 201us) server_negotiation.cc:407] Connection header received
0619 11:10:21.983898 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:21.983904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:21.983963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:21.984061 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:21.984585 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.985098 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.985842 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:21.986055 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:21.986132 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:21.988624 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:47401: BlockingRecv error: recv got EOF from 127.7.90.66:47401 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:22.188252 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:22.182822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:34739)
0619 11:10:22.183082 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:22.183095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:22.183202 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:22.183453 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:22.183455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:22.183639 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:22.183738 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.183742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.184506 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.184509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.185089 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.185099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.185543 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.185547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.185614 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.186243 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:22.186256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:22.188126 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260619 11:10:22.188388 16692 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:22.182874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:34739 (local address 127.7.90.126:45411)
0619 11:10:22.183007 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:22.183011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:22.183186 (+ 175us) server_negotiation.cc:407] Connection header received
0619 11:10:22.183292 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:22.183294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:22.183342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:22.183424 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:22.183845 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.184369 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.185204 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.185414 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.185493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.188268 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:34739: BlockingRecv error: recv got EOF from 127.7.90.67:34739 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:22.814117 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:22.808635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:39837)
0619 11:10:22.808874 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:22.808887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:22.808984 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:22.809240 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:22.809242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:22.809406 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:22.809489 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.809493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.810162 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.810165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.811025 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.811037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.811549 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.811553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.811618 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.812143 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:22.812156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:22.813934 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
W20260619 11:10:22.814201 16696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:22.808698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:39837 (local address 127.7.90.126:45411)
0619 11:10:22.808920 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:22.808923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:22.808934 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:10:22.809074 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:22.809076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:22.809117 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:22.809190 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:22.809604 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.810067 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.811148 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.811453 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.811531 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.814072 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:39837: BlockingRecv error: recv got EOF from 127.7.90.65:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":52,"threads_started":1}
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:22 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:22.995610 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:22.989441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:58611)
0619 11:10:22.989799 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:22.989816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:22.989923 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:22.990215 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:22.990217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:22.990425 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:22.990514 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.990519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.991262 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.991268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.991963 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:22.991974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.992612 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.992616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:22.992726 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.993391 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:22.993404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:22.995492 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
W20260619 11:10:22.995846 16696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:22.989634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:58611 (local address 127.7.90.126:45411)
0619 11:10:22.989801 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:22.989807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:22.989873 (+ 66us) server_negotiation.cc:407] Connection header received
0619 11:10:22.990055 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:22.990058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:22.990110 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:22.990196 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:22.990625 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.991155 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.992113 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:22.992434 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:22.992535 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:22.995618 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:58611: BlockingRecv error: recv got EOF from 127.7.90.66:58611 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:23.195067 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:23.188986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:46147)
0619 11:10:23.189299 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:23.189312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:23.189441 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:23.189748 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:23.189751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:23.189957 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:23.190045 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.190051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.190800 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.190805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.191769 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.191779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.192400 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.192403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.192467 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:23.193066 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:23.193080 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:23.194947 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
W20260619 11:10:23.195204 16696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:23.189110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:46147 (local address 127.7.90.126:45411)
0619 11:10:23.189257 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:23.189261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:23.189384 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:10:23.189586 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:23.189589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:23.189641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:23.189727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:23.190160 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.190679 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.191931 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.192293 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.192394 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:23.195079 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:46147: BlockingRecv error: recv got EOF from 127.7.90.67:46147 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:23.820416 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:23.814894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:49213)
0619 11:10:23.815180 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:23.815193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:23.815291 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:23.815520 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:23.815522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:23.815703 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:23.815793 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.815798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.816546 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.816549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.817175 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.817185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.817752 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.817755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.817821 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:23.818358 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:23.818369 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:23.820240 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
W20260619 11:10:23.820546 16700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:23.814934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:49213 (local address 127.7.90.126:45411)
0619 11:10:23.815195 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:23.815199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:23.815259 (+ 60us) server_negotiation.cc:407] Connection header received
0619 11:10:23.815367 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:23.815369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:23.815411 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:23.815498 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:23.815904 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.816443 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.817289 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.817534 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.817610 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:23.820384 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:49213: BlockingRecv error: recv got EOF from 127.7.90.65:49213 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:23 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:24.001286 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:23.996318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:36723)
0619 11:10:23.996561 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:23.996573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:23.996667 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:23.996889 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:23.996891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:23.997065 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:23.997156 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.997160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.997872 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.997874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.998421 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:23.998429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.998817 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.998820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:23.998876 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:23.999392 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:23.999403 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:24.001116 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
W20260619 11:10:24.001325 16700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:23.996357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36723 (local address 127.7.90.126:45411)
0619 11:10:23.996491 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:23.996495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:23.996639 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:10:23.996744 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:23.996746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:23.996788 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:23.996854 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:23.997248 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.997779 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.998512 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:23.998730 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:23.998800 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:24.001230 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:36723: BlockingRecv error: recv got EOF from 127.7.90.66:36723 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:10:24 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:24 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:24.201051 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:24.195880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:40751)
0619 11:10:24.196146 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:24.196161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:24.196267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:24.196490 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:24.196492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:24.196665 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:24.196756 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:24.196760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.197492 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.197495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:24.198112 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:24.198121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.198506 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.198509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:24.198568 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:24.199113 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:24.199124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:24.200930 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
W20260619 11:10:24.201179 16700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:24.195924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:40751 (local address 127.7.90.126:45411)
0619 11:10:24.196062 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:24.196065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:24.196238 (+ 173us) server_negotiation.cc:407] Connection header received
0619 11:10:24.196341 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:24.196343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:24.196386 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:24.196453 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:24.196866 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.197401 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.198211 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.198417 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.198489 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:24.201063 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:40751: BlockingRecv error: recv got EOF from 127.7.90.67:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 11:10:24 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:24 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:24.826719 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:24.821354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:41177)
0619 11:10:24.821694 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:24.821708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:24.821802 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:24.822068 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:24.822070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:24.822244 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:24.822329 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:24.822334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.823090 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.823092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:24.823633 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:24.823639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.824079 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.824081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:24.824138 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:24.824687 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:24.824698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:24.826550 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":125,"threads_started":1}
W20260619 11:10:24.826819 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:24.821404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:41177 (local address 127.7.90.126:45411)
0619 11:10:24.821690 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:24.821694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:24.821781 (+ 87us) server_negotiation.cc:407] Connection header received
0619 11:10:24.821886 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:24.821889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:24.821936 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:24.822024 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:24.822449 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.822990 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.823738 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:24.823975 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:24.824062 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:24.826681 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:41177: BlockingRecv error: recv got EOF from 127.7.90.65:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:25.007902 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:25.002277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:35201)
0619 11:10:25.002678 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:25.002696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:25.002829 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:25.003080 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:25.003083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:25.003295 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:25.003381 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.003386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.004137 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.004143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.004707 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.004714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.005160 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.005163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.005248 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.005762 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:25.005775 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:25.007708 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":106,"threads_started":1}
W20260619 11:10:25.007947 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:25.002355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:35201 (local address 127.7.90.126:45411)
0619 11:10:25.002525 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:25.002530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:25.002794 (+ 264us) server_negotiation.cc:407] Connection header received
0619 11:10:25.002929 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:25.002932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:25.002974 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:25.003064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:25.003507 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.004030 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.004826 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.005049 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.005157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.007836 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:35201: BlockingRecv error: recv got EOF from 127.7.90.66:35201 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:25.207207 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:25.201830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:40203)
0619 11:10:25.202163 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:25.202177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:25.202320 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:25.202591 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:25.202593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:25.202788 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:25.202878 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.202882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.203634 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.203636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.204187 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.204193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.204606 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.204609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.204671 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.205245 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:25.205258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:25.207091 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":89,"threads_started":1}
W20260619 11:10:25.207381 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:25.201909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:40203 (local address 127.7.90.126:45411)
0619 11:10:25.202065 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:25.202072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:25.202283 (+ 211us) server_negotiation.cc:407] Connection header received
0619 11:10:25.202396 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:25.202399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:25.202466 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:25.202551 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:25.202971 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.203543 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.204281 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.204509 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.204581 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.207225 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:40203: BlockingRecv error: recv got EOF from 127.7.90.67:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:25 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:25.835924 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:25.827461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:60367)
0619 11:10:25.827742 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:25.827755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:25.827843 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:25.828068 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:25.828070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:25.828252 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:25.828338 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.828343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.829015 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.829017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.829571 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:25.829578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.829969 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.829971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:25.830044 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.833819 (+ 3775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:25.833831 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:25.835737 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
W20260619 11:10:25.836009 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:25.827649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:60367 (local address 127.7.90.126:45411)
0619 11:10:25.827865 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:25.827869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:25.827893 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:10:25.827927 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:25.827929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:25.827967 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:25.828035 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:25.828430 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.828931 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.829667 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:25.829873 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:25.829954 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:25.835884 (+ 5930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:60367: BlockingRecv error: recv got EOF from 127.7.90.65:60367 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:26.014249 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:26.008730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:36777)
0619 11:10:26.009009 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:26.009023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:26.009123 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:26.009355 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:26.009357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:26.009596 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:26.009693 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.009699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.010559 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.010561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.011113 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.011120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.011537 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.011539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.011598 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.012140 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:26.012196 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:26.014128 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
W20260619 11:10:26.014384 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:26.008877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36777 (local address 127.7.90.126:45411)
0619 11:10:26.009034 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:26.009038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:26.009077 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:10:26.009205 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:26.009207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:26.009248 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:26.009322 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:26.009795 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.010468 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.011218 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.011440 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.011519 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.014264 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:36777: BlockingRecv error: recv got EOF from 127.7.90.66:36777 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:26.213008 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:26.207994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:35761)
0619 11:10:26.208254 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:26.208267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:26.208360 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:26.208579 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:26.208581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:26.208749 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:26.208857 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.208862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.209592 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.209595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.210130 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.210136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.210525 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.210528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.210587 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.211133 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:26.211145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:26.212899 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
W20260619 11:10:26.213137 16708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:26.208040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:35761 (local address 127.7.90.126:45411)
0619 11:10:26.208177 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:26.208182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:26.208336 (+ 154us) server_negotiation.cc:407] Connection header received
0619 11:10:26.208437 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:26.208439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:26.208483 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:26.208547 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:26.208948 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.209453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.210226 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.210429 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.210500 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.213018 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:35761: BlockingRecv error: recv got EOF from 127.7.90.67:35761 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:26 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:26.841754 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:26.836731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:53273)
0619 11:10:26.836993 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:26.837006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:26.837101 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:26.837330 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:26.837332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:26.837501 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:26.837665 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.837671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.838391 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.838394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.838920 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:26.838927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.839319 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.839321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:26.839379 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.839835 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:26.839845 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:26.841644 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
W20260619 11:10:26.841929 16712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:26.836779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:53273 (local address 127.7.90.126:45411)
0619 11:10:26.837016 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:26.837019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:26.837056 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:10:26.837174 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:26.837176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:26.837217 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:26.837295 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:26.837801 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.838306 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.839019 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:26.839221 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:26.839292 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:26.841762 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:53273: BlockingRecv error: recv got EOF from 127.7.90.65:53273 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:27.020601 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:27.015143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:34555)
0619 11:10:27.015403 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:27.015415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:27.015510 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:27.015722 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:27.015724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:27.015895 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:27.015987 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.015993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.016743 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.016745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.017296 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.017303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.017764 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.017790 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.017876 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.018313 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:27.018323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:27.020430 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
W20260619 11:10:27.020634 16712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:27.015206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:34555 (local address 127.7.90.126:45411)
0619 11:10:27.015341 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:27.015344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:27.015488 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:10:27.015575 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:27.015577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:27.015619 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:27.015691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:27.016078 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.016642 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.017388 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.017666 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.017741 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.020538 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:34555: BlockingRecv error: recv got EOF from 127.7.90.66:34555 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:27.219786 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:27.213913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:50555)
0619 11:10:27.214178 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:27.214191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:27.214285 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:27.214510 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:27.214513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:27.214683 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:27.214774 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.214779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.215751 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.215756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.216533 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.216543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.217005 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.217009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.217100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.217726 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:27.217739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:27.219675 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
W20260619 11:10:27.219871 16712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:27.214038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:50555 (local address 127.7.90.126:45411)
0619 11:10:27.214171 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:27.214177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:27.214256 (+ 79us) server_negotiation.cc:407] Connection header received
0619 11:10:27.214360 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:27.214362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:27.214405 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:27.214478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:27.214864 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.215634 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.216632 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.216835 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.216927 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.219785 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:50555: BlockingRecv error: recv got EOF from 127.7.90.67:50555 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:27 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:27.848809 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:27.842534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:43965)
0619 11:10:27.842845 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:27.842861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:27.842984 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:27.843257 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:27.843261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:27.843459 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:27.843557 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.843563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.844349 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.844354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.844948 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:27.844957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.845415 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.845418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:27.845546 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.846104 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:27.846136 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:27.848643 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
W20260619 11:10:27.848882 16716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:27.842586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:43965 (local address 127.7.90.126:45411)
0619 11:10:27.842861 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:27.842865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:27.842949 (+ 84us) server_negotiation.cc:407] Connection header received
0619 11:10:27.843082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:27.843085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:27.843128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:27.843231 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:27.843678 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.844225 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.845072 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:27.845302 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:27.845409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:27.848762 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:43965: BlockingRecv error: recv got EOF from 127.7.90.65:43965 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:28.026968 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:28.021344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:36109)
0619 11:10:28.021632 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:28.021647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:28.021745 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:28.021973 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:28.021975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:28.022162 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:28.022260 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.022266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.022980 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.022984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.023843 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.023854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.024377 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.024380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.024439 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.024984 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:28.024998 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:28.026847 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260619 11:10:28.027143 16716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:28.021418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36109 (local address 127.7.90.126:45411)
0619 11:10:28.021594 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:28.021598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:28.021714 (+ 116us) server_negotiation.cc:407] Connection header received
0619 11:10:28.021823 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:28.021825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:28.021866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:28.021934 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:28.022358 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.022868 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.023976 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.024276 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.024364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.026990 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:36109: BlockingRecv error: recv got EOF from 127.7.90.66:36109 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:28.226141 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:28.220554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:48065)
0619 11:10:28.220877 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:28.220892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:28.221016 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:28.221292 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:28.221296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:28.221509 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:28.221625 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.221631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.222372 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.222377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.222946 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.222953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.223403 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.223407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.223494 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.224040 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:28.224061 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:28.225978 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
W20260619 11:10:28.226204 16716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:28.220701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:48065 (local address 127.7.90.126:45411)
0619 11:10:28.220846 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:28.220851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:28.220969 (+ 118us) server_negotiation.cc:407] Connection header received
0619 11:10:28.221116 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:28.221120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:28.221173 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:28.221257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:28.221736 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.222266 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.223068 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.223299 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.223403 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.226103 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:48065: BlockingRecv error: recv got EOF from 127.7.90.67:48065 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:28 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:28.854693 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:28.849557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:59037)
0619 11:10:28.849854 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:28.849867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:28.849955 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:28.850170 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:28.850172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:28.850342 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:28.850430 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.850435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.851111 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.851113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.851670 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:28.851677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.852227 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.852231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:28.852293 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.852744 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:28.852755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:28.854525 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
W20260619 11:10:28.854748 16720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:28.849630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:59037 (local address 127.7.90.126:45411)
0619 11:10:28.849893 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:28.849896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:28.849912 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:10:28.850021 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:28.850023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:28.850063 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:28.850137 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:28.850526 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.851026 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.851763 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:28.852001 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:28.852192 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:28.854635 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:59037: BlockingRecv error: recv got EOF from 127.7.90.65:59037 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:29.033668 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:29.027782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:46111)
0619 11:10:29.028076 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:29.028090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:29.028241 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:29.028506 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:29.028509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:29.028722 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:29.028829 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.028835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.029824 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.029828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.030704 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.030721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.031121 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.031124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.031181 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.031684 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:29.031696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:29.033455 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
W20260619 11:10:29.033775 16720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:29.027860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:46111 (local address 127.7.90.126:45411)
0619 11:10:29.028022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:29.028026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:29.028194 (+ 168us) server_negotiation.cc:407] Connection header received
0619 11:10:29.028353 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:29.028357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:29.028405 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:29.028496 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:29.028951 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.029653 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.030820 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.031019 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.031091 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.033668 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:46111: BlockingRecv error: recv got EOF from 127.7.90.66:46111 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:29.231792 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:29.226861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:47863)
0619 11:10:29.227124 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:29.227137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:29.227231 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:29.227471 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:29.227473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:29.227646 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:29.227739 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.227744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.228429 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.228431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.228971 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.228977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.229364 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.229366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.229424 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.229935 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:29.229947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:29.231660 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
W20260619 11:10:29.231870 16720 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:29.226917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:47863 (local address 127.7.90.126:45411)
0619 11:10:29.227040 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:29.227043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:29.227200 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:10:29.227308 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:29.227310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:29.227369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:29.227435 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:29.227828 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.228344 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.229077 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.229277 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.229347 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.231781 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:47863: BlockingRecv error: recv got EOF from 127.7.90.67:47863 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:29 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:29.861094 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:29.855511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:53321)
0619 11:10:29.855797 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:29.855821 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:29.855942 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:29.856186 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:29.856188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:29.856375 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:29.856475 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.856481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.857361 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.857367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.858238 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:29.858250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.858654 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.858656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:29.858712 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.859239 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:29.859253 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:29.860982 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
W20260619 11:10:29.861255 16724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:29.855600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:53321 (local address 127.7.90.126:45411)
0619 11:10:29.855880 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:29.855883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:29.855896 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:29.856026 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:29.856029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:29.856070 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:29.856148 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:29.856580 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.857229 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.858341 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:29.858563 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:29.858642 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:29.861090 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:53321: BlockingRecv error: recv got EOF from 127.7.90.65:53321 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":134,"threads_started":1}
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:30.040053 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:30.034534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:58423)
0619 11:10:30.034853 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:30.034875 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:30.035013 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:30.035280 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:30.035283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:30.035491 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:30.035578 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.035583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.036353 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.036357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.036909 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.036917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.037371 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.037374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.037462 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.037967 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:30.037978 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:30.039870 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":83,"threads_started":1}
W20260619 11:10:30.040082 16724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:30.034669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:58423 (local address 127.7.90.126:45411)
0619 11:10:30.034821 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:30.034825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:30.034954 (+ 129us) server_negotiation.cc:407] Connection header received
0619 11:10:30.035112 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:30.035117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:30.035170 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:30.035258 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:30.035700 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.036232 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.037027 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.037258 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.037368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.039972 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:58423: BlockingRecv error: recv got EOF from 127.7.90.66:58423 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:30.238438 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:30.232670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:57829)
0619 11:10:30.232963 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:30.232976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:30.233119 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:30.233387 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:30.233390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:30.233633 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:30.233723 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.233729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.234475 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.234481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.235140 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.235148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.235678 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.235682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.235801 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.236386 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:30.236408 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:30.238272 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
W20260619 11:10:30.238473 16724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:30.232735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:57829 (local address 127.7.90.126:45411)
0619 11:10:30.232898 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:30.232903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:30.233050 (+ 147us) server_negotiation.cc:407] Connection header received
0619 11:10:30.233233 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:30.233235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:30.233279 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:30.233364 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:30.233843 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.234364 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.235268 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.235527 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.235645 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.238379 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:57829: BlockingRecv error: recv got EOF from 127.7.90.67:57829 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:30 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:30.872846 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:30.861967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:38387)
0619 11:10:30.862348 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:30.862364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:30.862480 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:30.862883 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:30.862886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:30.863126 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:30.863270 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.863281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.864133 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.864138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.864704 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:30.864712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.865167 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.865171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:30.865254 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.870914 (+ 5660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:30.870925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:30.872677 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":103,"threads_started":1}
W20260619 11:10:30.872921 16728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:30.862107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:38387 (local address 127.7.90.126:45411)
0619 11:10:30.862349 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:30.862355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:30.862425 (+ 70us) server_negotiation.cc:407] Connection header received
0619 11:10:30.862597 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:30.862601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:30.862655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:30.862755 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:30.863373 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.864025 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.864824 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:30.865056 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:30.865166 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:30.872797 (+ 7631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:38387: BlockingRecv error: recv got EOF from 127.7.90.65:38387 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":57,"threads_started":1}
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:31.046001 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:31.041029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:33379)
0619 11:10:31.041336 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:31.041349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:31.041439 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:31.041674 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:31.041676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:31.041847 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:31.041935 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.041940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.042647 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.042649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.043211 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.043218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.043602 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.043604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.043673 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.044145 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:31.044157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:31.045857 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
W20260619 11:10:31.046083 16728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:31.041171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:33379 (local address 127.7.90.126:45411)
0619 11:10:31.041327 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:31.041330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:31.041397 (+ 67us) server_negotiation.cc:407] Connection header received
0619 11:10:31.041504 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:31.041506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:31.041563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:31.041635 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:31.042058 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.042560 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.043306 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.043511 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.043583 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.045984 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:33379: BlockingRecv error: recv got EOF from 127.7.90.66:33379 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:31.244354 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:31.239207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:46841)
0619 11:10:31.239470 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:31.239482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:31.239576 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:31.239824 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:31.239826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:31.240011 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:31.240102 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.240107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.240784 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.240786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.241372 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.241379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.241787 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.241790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.241851 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.242440 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:31.242452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:31.244237 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260619 11:10:31.244558 16728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:31.239292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:46841 (local address 127.7.90.126:45411)
0619 11:10:31.239418 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:31.239422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:31.239548 (+ 126us) server_negotiation.cc:407] Connection header received
0619 11:10:31.239651 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:31.239654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:31.239696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:31.239767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:31.240193 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.240681 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.241467 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.241696 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.241769 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.244349 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:46841: BlockingRecv error: recv got EOF from 127.7.90.67:46841 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:31 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:31.878512 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:31.873584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:56867)
0619 11:10:31.873859 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:31.873872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:31.873960 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:31.874207 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:31.874209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:31.874381 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:31.874470 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.874487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.875192 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.875194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.875723 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:31.875730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.876142 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.876145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:31.876206 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.876693 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:31.876704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:31.878390 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260619 11:10:31.878764 16732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:31.873640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:56867 (local address 127.7.90.126:45411)
0619 11:10:31.873925 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:31.873928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:31.873941 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:31.874053 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:31.874056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:31.874099 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:31.874172 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:31.874574 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.875109 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.875831 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:31.876041 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:31.876120 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:31.878513 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:56867: BlockingRecv error: recv got EOF from 127.7.90.65:56867 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":59,"threads_started":1}
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:32.051667 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:32.046797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:55969)
0619 11:10:32.047054 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:32.047067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:32.047158 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:32.047372 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:32.047374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:32.047543 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:32.047632 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.047636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.048302 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.048304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.048837 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.048844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.049217 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.049219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.049274 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.049867 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:32.049878 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:32.051546 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
W20260619 11:10:32.051836 16732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:32.046909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:55969 (local address 127.7.90.126:45411)
0619 11:10:32.047053 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:32.047057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:32.047131 (+ 74us) server_negotiation.cc:407] Connection header received
0619 11:10:32.047231 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:32.047233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:32.047274 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:32.047338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:32.047720 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.048215 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.048929 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.049127 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.049196 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.051667 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:55969: BlockingRecv error: recv got EOF from 127.7.90.66:55969 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.67@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:32.250560 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:32.245055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.67:48929)
0619 11:10:32.245341 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:32.245353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:32.245474 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:32.245811 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:32.245814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:32.246029 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:32.246120 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.246126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.246867 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.246872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.247489 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.247496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.247943 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.247946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.248025 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.248544 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:32.248565 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:32.250397 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
W20260619 11:10:32.250607 16732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:32.245115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:48929 (local address 127.7.90.126:45411)
0619 11:10:32.245273 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:32.245278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:32.245423 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:10:32.245648 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:32.245651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:32.245701 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:32.245789 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:32.246231 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.246757 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.247617 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.247841 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.247941 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.250500 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.67:48929: BlockingRecv error: recv got EOF from 127.7.90.67:48929 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:32 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.65@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:32.884678 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:32.879481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.65:39113)
0619 11:10:32.879737 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:32.879750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:32.879839 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:32.880074 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:32.880077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:32.880276 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:32.880367 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.880372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.881075 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.881077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.881687 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:32.881695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.882172 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.882176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:32.882261 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.882725 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:32.882736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:32.884487 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
W20260619 11:10:32.884742 16736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:32.879529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:39113 (local address 127.7.90.126:45411)
0619 11:10:32.879759 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:32.879763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:32.879797 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:10:32.879918 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:32.879920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:32.879962 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:32.880045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:32.880463 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.880984 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.881813 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:32.882053 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:32.882168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:32.884621 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.65:39113: BlockingRecv error: recv got EOF from 127.7.90.65:39113 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
Jun 19 11:10:33 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:10:33 dist-test-slave-f1t5 krb5kdc[15678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.90.66@KRBTEST.COM for oryx/127.7.90.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:10:33.057803 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.052589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45411 (local address 127.7.90.66:34517)
0619 11:10:33.052846 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:33.052859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:33.052947 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:33.053188 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:33.053190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:33.053362 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:33.053452 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.053457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.054167 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.054170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.054720 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.054727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.055110 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.055113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.055173 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.055703 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:33.055715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:33.057634 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.90.126:45411: Server oryx/127.7.90.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260619 11:10:33.058033 16736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:33.052693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:34517 (local address 127.7.90.126:45411)
0619 11:10:33.052832 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:33.052841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:33.052928 (+ 87us) server_negotiation.cc:407] Connection header received
0619 11:10:33.053039 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:33.053041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:33.053083 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:33.053154 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:33.053557 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.054068 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.054818 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.055021 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.055091 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.057781 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.90.66:34517: BlockingRecv error: recv got EOF from 127.7.90.66:34517 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:10:33.149853 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16261
I20260619 11:10:33.155511 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16413
I20260619 11:10:33.160576 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16545
I20260619 11:10:33.165747 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16191
2026-06-19T11:10:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17961 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T11:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:33Z Disabled control of system clock
I20260619 11:10:33.195633 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:39535
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:10:33.301851 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:33.302126 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:33.302160 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:33.306066 16741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:33.306124 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:33.306140 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:33.306156 16741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:33.306171 16741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:33.310781 16741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:39535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:33.311755 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:33.313040 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:33.321587 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.321681 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.321732 16741 server_base.cc:1034] running on GCE node
W20260619 11:10:33.321822 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.322419 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:33.323828 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:33.325004 16741 hybrid_clock.cc:648] HybridClock initialized: now 1781867433324990 us; error 25 us; skew 500 ppm
I20260619 11:10:33.326689 16741 webserver.cc:460] Webserver started at http://127.7.90.126:34855/ using document root <none> and password file <none>
I20260619 11:10:33.327273 16741 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:33.327363 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:33.327593 16741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:33.329500 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "ed2c903fe1fe456dabe4f9f74798c72f"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "3e2769fffeef0bbdfaa1f9c5627efec4"
server_key_iv: "19aefc15c7fd237a13962ecfa48a5ad5"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.330083 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "ed2c903fe1fe456dabe4f9f74798c72f"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "3e2769fffeef0bbdfaa1f9c5627efec4"
server_key_iv: "19aefc15c7fd237a13962ecfa48a5ad5"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.333588 16741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:33.335729 16755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.336796 16741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:10:33.336933 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "ed2c903fe1fe456dabe4f9f74798c72f"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "3e2769fffeef0bbdfaa1f9c5627efec4"
server_key_iv: "19aefc15c7fd237a13962ecfa48a5ad5"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.337039 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:33.357542 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:33.358162 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:33.358346 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:33.365094 16741 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:39535
I20260619 11:10:33.365098 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:39535 every 8 connection(s)
I20260619 11:10:33.365933 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:33.368865 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:33.371584 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16741
I20260619 11:10:33.371673 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:33.371958 7529 external_mini_cluster.cc:1428] Setting key 140d43d5d4c52197d08bd3ef4854d4ee
I20260619 11:10:33.374394 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f: Bootstrap starting.
I20260619 11:10:33.376420 16808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:33.377112 16808 log.cc:824] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:33.379029 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f: No bootstrap required, opened a new log
I20260619 11:10:33.380698 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.373265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.7.90.126:39535)
0619 11:10:33.373908 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:33.373922 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:33.373955 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:10:33.374591 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:33.374608 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:33.374971 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:33.375287 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:33.375695 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.376665 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.377653 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.378111 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.378274 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.379157 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:33.379162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:33.379173 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:33.379519 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:33.379574 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:33.379592 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:33.379696 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:33.379836 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:10:33.380078 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":162,"threads_started":1}
I20260619 11:10:33.382730 16808 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } }
I20260619 11:10:33.383023 16808 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:33.383087 16808 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2c903fe1fe456dabe4f9f74798c72f, State: Initialized, Role: FOLLOWER
I20260619 11:10:33.383422 16808 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } }
I20260619 11:10:33.383536 16808 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:33.383581 16808 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:33.383646 16808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:33.384405 16808 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } }
I20260619 11:10:33.384672 16808 leader_election.cc:302] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2c903fe1fe456dabe4f9f74798c72f; no voters:
I20260619 11:10:33.384893 16808 leader_election.cc:288] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:33.384985 16813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:33.385264 16813 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [term 1 LEADER]: Becoming Leader. State: Replica: ed2c903fe1fe456dabe4f9f74798c72f, State: Running, Role: LEADER
I20260619 11:10:33.385661 16813 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } }
I20260619 11:10:33.386212 16808 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:33.387189 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2c903fe1fe456dabe4f9f74798c72f. Latest consensus state: current_term: 1 leader_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } } }
I20260619 11:10:33.387313 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:33.387669 16822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:33.387624 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c903fe1fe456dabe4f9f74798c72f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39535 } } }
I20260619 11:10:33.387760 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:33.390831 16822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:33.395764 16822 catalog_manager.cc:1347] Generated new cluster ID: 514cc725b4434d21b65224404631310b
I20260619 11:10:33.395825 16822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:33.417960 16822 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:33.418921 16822 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:33.423992 16822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2c903fe1fe456dabe4f9f74798c72f: Generated new TSK 0
I20260619 11:10:33.424691 16822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:33.433244 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--builtin_ntp_servers=127.7.90.84:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 11:10:33.542912 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:33.543154 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:33.543217 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:33.547152 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:33.547202 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:33.547253 16832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:33.551784 16832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:33.552872 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:33.554267 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:33.563478 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.563472 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.563766 16832 server_base.cc:1034] running on GCE node
W20260619 11:10:33.563485 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.564319 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:33.564957 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:33.566138 16832 hybrid_clock.cc:648] HybridClock initialized: now 1781867433566127 us; error 34 us; skew 500 ppm
I20260619 11:10:33.567893 16832 webserver.cc:460] Webserver started at http://127.7.90.65:40833/ using document root <none> and password file <none>
I20260619 11:10:33.568480 16832 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:33.568567 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:33.568791 16832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:33.570897 16832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "ed6566820a2045adb687ac2ee1b6e61b"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "8f71a8aacdb1b2e610a427cb26fdd074"
server_key_iv: "24fcb9ae51c6c0753091a638f58d03c0"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.571465 16832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "ed6566820a2045adb687ac2ee1b6e61b"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "8f71a8aacdb1b2e610a427cb26fdd074"
server_key_iv: "24fcb9ae51c6c0753091a638f58d03c0"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.575088 16832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:10:33.577639 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.578768 16832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:10:33.578898 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "ed6566820a2045adb687ac2ee1b6e61b"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "8f71a8aacdb1b2e610a427cb26fdd074"
server_key_iv: "24fcb9ae51c6c0753091a638f58d03c0"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.579010 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:33.598105 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:33.598804 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:33.599011 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:33.599609 16832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:33.600586 16832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:33.600661 16832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.600730 16832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:33.600780 16832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.609966 16832 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:45551
I20260619 11:10:33.610030 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:45551 every 8 connection(s)
I20260619 11:10:33.610906 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:33.619796 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16832
I20260619 11:10:33.619948 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:33.620039 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.612693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:47131 (local address 127.7.90.126:39535)
0619 11:10:33.612816 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:33.612820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:33.613917 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 11:10:33.615301 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:33.615305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:33.615361 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:33.615465 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:33.616338 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.616879 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.618132 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.618359 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.618483 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.619531 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:33.619535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:33.619538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:33.619558 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:33.619574 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:33.619582 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:33.619662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:33.619830 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 11:10:33.619884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:10:33.620334 7529 external_mini_cluster.cc:1428] Setting key a55b8280e79b98cc3a8e0de10cd7fa5e
I20260619 11:10:33.620935 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.613060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39535 (local address 127.7.90.65:47131)
0619 11:10:33.613777 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:33.613817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:33.615082 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:33.615642 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:33.615648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:33.615733 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:33.616199 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.616211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.617011 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.617021 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.618003 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.618013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.618516 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.618520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.618713 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.619270 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:33.619272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:33.619308 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:33.619366 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:33.619664 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:33.619675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:33.619683 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:10:33.619819 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 11:10:33.620095 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260619 11:10:33.622138 16960 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39535
I20260619 11:10:33.622327 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--builtin_ntp_servers=127.7.90.84:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:10:33.622372 16960 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:33.622937 16960 heartbeater.cc:511] Master 127.7.90.126:39535 requested a full tablet report, sending...
I20260619 11:10:33.624394 16771 ts_manager.cc:194] Registered new tserver with Master: ed6566820a2045adb687ac2ee1b6e61b (127.7.90.65:45551)
I20260619 11:10:33.625505 16771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.65:47131
W20260619 11:10:33.730234 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:33.730516 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:33.730587 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:33.734496 16963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:33.734575 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:33.734664 16963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:33.739087 16963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:33.740089 16963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:33.741456 16963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:33.751245 16969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.751322 16971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.751315 16968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.751534 16963 server_base.cc:1034] running on GCE node
I20260619 11:10:33.752126 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:33.752940 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:33.754163 16963 hybrid_clock.cc:648] HybridClock initialized: now 1781867433754115 us; error 67 us; skew 500 ppm
I20260619 11:10:33.756146 16963 webserver.cc:460] Webserver started at http://127.7.90.66:39769/ using document root <none> and password file <none>
I20260619 11:10:33.756778 16963 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:33.756875 16963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:33.757114 16963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:33.759303 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "6632f82053ba4e6eafab48017a33b99d"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "a3297c8157a63b5e2a07e3a1eb8b87bb"
server_key_iv: "a89792396a8af6305aebfd064252e29d"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.759917 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "6632f82053ba4e6eafab48017a33b99d"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "a3297c8157a63b5e2a07e3a1eb8b87bb"
server_key_iv: "a89792396a8af6305aebfd064252e29d"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.763826 16963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 11:10:33.766253 16977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.767370 16963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:33.767508 16963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "6632f82053ba4e6eafab48017a33b99d"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "a3297c8157a63b5e2a07e3a1eb8b87bb"
server_key_iv: "a89792396a8af6305aebfd064252e29d"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.767616 16963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:33.796332 16963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:33.797089 16963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:33.797295 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:33.797962 16963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:33.798918 16963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:33.798992 16963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.799064 16963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:33.799098 16963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.807546 16963 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:34329
I20260619 11:10:33.807578 17090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:34329 every 8 connection(s)
I20260619 11:10:33.808432 16963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:33.808884 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 16963
I20260619 11:10:33.808993 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:33.809325 7529 external_mini_cluster.cc:1428] Setting key 890356ab7d8c1174002dc98bc1a1ad91
I20260619 11:10:33.811717 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--builtin_ntp_servers=127.7.90.84:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:10:33.819105 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:45181 (local address 127.7.90.126:39535)
0619 11:10:33.810606 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:33.810612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:33.811893 (+ 1281us) server_negotiation.cc:407] Connection header received
0619 11:10:33.813610 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:33.813628 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:33.813671 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:33.813741 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:33.814703 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.815213 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.817076 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.817279 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.817351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.818659 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:33.818662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:33.818664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:33.818675 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:33.818693 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:33.818700 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:33.818757 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:33.818944 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 11:10:33.818989 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:10:33.820482 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:33.811024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39535 (local address 127.7.90.66:45181)
0619 11:10:33.811733 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:33.811772 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:33.813326 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:33.813928 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:33.813938 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:33.814054 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:33.814585 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.814598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.815343 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.815350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.816929 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:33.816956 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:33.817433 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:33.817438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:33.817702 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:33.818350 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:33.818353 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:33.818415 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:33.818504 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:33.818786 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:33.818798 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:33.818805 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:10:33.818959 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 11:10:33.819380 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260619 11:10:33.821743 17091 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39535
I20260619 11:10:33.822041 17091 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:33.822678 17091 heartbeater.cc:511] Master 127.7.90.126:39535 requested a full tablet report, sending...
I20260619 11:10:33.823784 16771 ts_manager.cc:194] Registered new tserver with Master: 6632f82053ba4e6eafab48017a33b99d (127.7.90.66:34329)
I20260619 11:10:33.824327 16771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.66:45181
W20260619 11:10:33.925562 17094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:33.925813 17094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:33.925876 17094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:33.929958 17094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:33.930007 17094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:33.930091 17094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:33.934682 17094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:33.935707 17094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:33.937036 17094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:33.946259 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.946324 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:33.946237 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:33.946559 17094 server_base.cc:1034] running on GCE node
I20260619 11:10:33.946942 17094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:33.947606 17094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:33.948786 17094 hybrid_clock.cc:648] HybridClock initialized: now 1781867433948775 us; error 28 us; skew 500 ppm
I20260619 11:10:33.950603 17094 webserver.cc:460] Webserver started at http://127.7.90.67:40031/ using document root <none> and password file <none>
I20260619 11:10:33.951169 17094 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:33.951256 17094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:33.951469 17094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:33.953426 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "c7914cdab55b4a7985dd01fb21c80b56"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "88d20dbcd18f3e84844f52cbc64b977b"
server_key_iv: "b0e6ce5f79b41f6223c8aac9dcb2c860"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.954098 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "c7914cdab55b4a7985dd01fb21c80b56"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "88d20dbcd18f3e84844f52cbc64b977b"
server_key_iv: "b0e6ce5f79b41f6223c8aac9dcb2c860"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.957616 17094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:10:33.959969 17108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.961155 17094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:10:33.961288 17094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "c7914cdab55b4a7985dd01fb21c80b56"
format_stamp: "Formatted at 2026-06-19 11:10:33 on dist-test-slave-f1t5"
server_key: "88d20dbcd18f3e84844f52cbc64b977b"
server_key_iv: "b0e6ce5f79b41f6223c8aac9dcb2c860"
server_key_version: "encryptionkey@0"
I20260619 11:10:33.961400 17094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:33.993173 17094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:33.993968 17094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:33.994174 17094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:33.994776 17094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:33.995738 17094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:33.995811 17094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:33.995903 17094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:33.995949 17094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.004556 17094 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:33565
I20260619 11:10:34.004575 17221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:33565 every 8 connection(s)
I20260619 11:10:34.005548 17094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:34.009006 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17094
I20260619 11:10:34.009099 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:34.009367 7529 external_mini_cluster.cc:1428] Setting key a2f82796fba514aeae6578e1ec61bd51
I20260619 11:10:34.014789 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.007386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:57163 (local address 127.7.90.126:39535)
0619 11:10:34.007517 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.007527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.008549 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 11:10:34.010038 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.010041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.010094 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.010200 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.011087 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:34.011640 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:34.012762 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:34.012987 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:34.013100 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:34.014138 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:34.014141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:34.014144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:34.014157 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:34.014179 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:34.014191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:34.014265 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:34.014489 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:10:34.014591 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:10:34.015758 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.007751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39535 (local address 127.7.90.67:57163)
0619 11:10:34.008396 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:34.008436 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:34.009834 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:34.010367 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:34.010373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:34.010455 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:34.010948 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:34.010960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:34.011765 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:34.011769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:34.012639 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:34.012647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:34.013143 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:34.013147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:34.013306 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:34.013881 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:34.013883 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:34.013923 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:34.013985 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:34.014287 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:34.014312 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:34.014318 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:10:34.014483 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 11:10:34.014834 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260619 11:10:34.016780 17222 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39535
I20260619 11:10:34.017030 17222 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:34.017581 17222 heartbeater.cc:511] Master 127.7.90.126:39535 requested a full tablet report, sending...
I20260619 11:10:34.018523 16771 ts_manager.cc:194] Registered new tserver with Master: c7914cdab55b4a7985dd01fb21c80b56 (127.7.90.67:33565)
I20260619 11:10:34.019026 16771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.67:57163
I20260619 11:10:34.023029 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:10:34.025759 16811 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:34.024769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.7.90.126:39535)
0619 11:10:34.024888 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.024893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.025097 (+ 204us) server_negotiation.cc:407] Connection header received
0619 11:10:34.025253 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.025256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.025302 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.025379 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.025644 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56506: BlockingRecv error: recv got EOF from 127.0.0.1:56506 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:34.025905 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16832
I20260619 11:10:34.031358 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16963
I20260619 11:10:34.036458 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17094
I20260619 11:10:34.041428 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 16741
2026-06-19T11:10:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (874 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T11:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:34Z Disabled control of system clock
I20260619 11:10:34.069695 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:39011
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:39011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:10:34.174733 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:34.174973 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:34.175007 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:34.178876 17233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:34.178928 17233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:34.178943 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:34.178958 17233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:34.178992 17233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:34.183525 17233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:39011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:39011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:34.184576 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:34.185954 17233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:34.194244 17241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.194427 17233 server_base.cc:1034] running on GCE node
W20260619 11:10:34.194343 17238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.194250 17239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.195082 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:34.196134 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:34.197321 17233 hybrid_clock.cc:648] HybridClock initialized: now 1781867434197302 us; error 31 us; skew 500 ppm
I20260619 11:10:34.199138 17233 webserver.cc:460] Webserver started at http://127.7.90.126:36469/ using document root <none> and password file <none>
I20260619 11:10:34.199692 17233 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:34.199785 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:34.200011 17233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:34.202144 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "963696b3e0914e8fa4f3017ba66c00f5"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "fb6bb043dbb469cf95e7846e1c29a39e"
server_key_iv: "c58c5e59dd3fe4ff2ad92d49651e6ff1"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.202679 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "963696b3e0914e8fa4f3017ba66c00f5"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "fb6bb043dbb469cf95e7846e1c29a39e"
server_key_iv: "c58c5e59dd3fe4ff2ad92d49651e6ff1"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.206620 17233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:10:34.209074 17247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.210318 17233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:34.210479 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "963696b3e0914e8fa4f3017ba66c00f5"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "fb6bb043dbb469cf95e7846e1c29a39e"
server_key_iv: "c58c5e59dd3fe4ff2ad92d49651e6ff1"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.210597 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:34.215210 17233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:34.215801 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:34.215991 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:34.222451 17233 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:39011
I20260619 11:10:34.222455 17299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:39011 every 8 connection(s)
I20260619 11:10:34.223340 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:34.224586 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17233
I20260619 11:10:34.224700 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:34.225005 7529 external_mini_cluster.cc:1428] Setting key d1419a69f19e43e5bfcdae44360389b4
I20260619 11:10:34.226881 17300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:34.230484 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.226419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.7.90.126:39011)
0619 11:10:34.227132 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.227148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.227181 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:10:34.227841 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.227857 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.228187 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.228532 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.229263 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:34.229267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:34.229278 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:34.229286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:34.229332 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:34.229347 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:34.229541 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:34.229615 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 11:10:34.229841 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":154,"threads_started":1}
I20260619 11:10:34.232870 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5: Bootstrap starting.
I20260619 11:10:34.234910 17300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:34.235606 17300 log.cc:824] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:34.237344 17300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5: No bootstrap required, opened a new log
I20260619 11:10:34.240899 17300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } }
I20260619 11:10:34.241096 17300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:34.241149 17300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963696b3e0914e8fa4f3017ba66c00f5, State: Initialized, Role: FOLLOWER
I20260619 11:10:34.241595 17300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } }
I20260619 11:10:34.241741 17300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:34.241804 17300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:34.241902 17300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:34.242795 17300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } }
I20260619 11:10:34.243117 17300 leader_election.cc:302] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963696b3e0914e8fa4f3017ba66c00f5; no voters:
I20260619 11:10:34.243395 17300 leader_election.cc:288] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:34.243672 17305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:34.243851 17305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [term 1 LEADER]: Becoming Leader. State: Replica: 963696b3e0914e8fa4f3017ba66c00f5, State: Running, Role: LEADER
I20260619 11:10:34.244163 17305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } }
I20260619 11:10:34.244556 17300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:34.245864 17306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963696b3e0914e8fa4f3017ba66c00f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } } }
I20260619 11:10:34.245864 17307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963696b3e0914e8fa4f3017ba66c00f5. Latest consensus state: current_term: 1 leader_uuid: "963696b3e0914e8fa4f3017ba66c00f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963696b3e0914e8fa4f3017ba66c00f5" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 39011 } } }
I20260619 11:10:34.246004 17307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:34.246004 17306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:34.246500 17317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:34.249058 17317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:34.254194 17317 catalog_manager.cc:1347] Generated new cluster ID: b4a6615d2ce24e8eadbffae063e73679
I20260619 11:10:34.254263 17317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:34.266640 17317 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:34.266911 17317 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:34.270340 17317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963696b3e0914e8fa4f3017ba66c00f5: Generated new TSK 0
I20260619 11:10:34.270913 17317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:10:34.275267 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--builtin_ntp_servers=127.7.90.84:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:10:34.400166 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:34.400417 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:34.400457 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:34.404498 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:34.404558 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:34.404615 17324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:34.409370 17324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:34.410526 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:34.411898 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:34.421406 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.421399 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.421399 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.421530 17324 server_base.cc:1034] running on GCE node
I20260619 11:10:34.422083 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:34.422705 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:34.423869 17324 hybrid_clock.cc:648] HybridClock initialized: now 1781867434423854 us; error 27 us; skew 500 ppm
I20260619 11:10:34.425709 17324 webserver.cc:460] Webserver started at http://127.7.90.65:36281/ using document root <none> and password file <none>
I20260619 11:10:34.426280 17324 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:34.426339 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:34.426577 17324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:34.428591 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "c99fc169ca74415dac6df1ebf860edec"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "b49748041f5252929d605f5fc9e68ad6"
server_key_iv: "cb5a3ae454397aae273c4b5dfce88beb"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.429139 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "c99fc169ca74415dac6df1ebf860edec"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "b49748041f5252929d605f5fc9e68ad6"
server_key_iv: "cb5a3ae454397aae273c4b5dfce88beb"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.432655 17324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:10:34.435052 17338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.436173 17324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:10:34.436286 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "c99fc169ca74415dac6df1ebf860edec"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "b49748041f5252929d605f5fc9e68ad6"
server_key_iv: "cb5a3ae454397aae273c4b5dfce88beb"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.436386 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:34.440757 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:34.441299 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:34.441474 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:34.441982 17324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:34.442837 17324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:34.442880 17324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.442945 17324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:34.442974 17324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.451489 17324 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:45727
I20260619 11:10:34.451509 17451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:45727 every 8 connection(s)
I20260619 11:10:34.452329 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:34.458031 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.454065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:33889 (local address 127.7.90.126:39011)
0619 11:10:34.454201 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.454205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.455248 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 11:10:34.456428 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.456431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.456465 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.456538 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.457624 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:34.457628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:34.457630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:34.457641 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:34.457665 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:34.457673 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:34.457757 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:34.457880 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 11:10:34.457916 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:10:34.458827 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.454435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39011 (local address 127.7.90.65:33889)
0619 11:10:34.455107 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:34.455146 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:34.456257 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:34.456694 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:34.456700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:34.456780 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:34.457360 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:34.457362 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:34.457400 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:34.457464 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:34.457760 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:34.457767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:34.457771 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:10:34.457900 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 11:10:34.458082 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260619 11:10:34.459987 17452 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39011
I20260619 11:10:34.460211 17452 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:34.460608 17452 heartbeater.cc:511] Master 127.7.90.126:39011 requested a full tablet report, sending...
I20260619 11:10:34.462013 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17324
I20260619 11:10:34.462153 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:34.462164 17264 ts_manager.cc:194] Registered new tserver with Master: c99fc169ca74415dac6df1ebf860edec (127.7.90.65:45727)
I20260619 11:10:34.462502 7529 external_mini_cluster.cc:1428] Setting key 9ebd622e357878b8b74a7575e3cca0fc
I20260619 11:10:34.464509 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--builtin_ntp_servers=127.7.90.84:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:10:34.572791 17455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:34.573055 17455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:34.573125 17455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:34.577117 17455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:34.577195 17455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:34.577282 17455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:34.581928 17455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:34.582973 17455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:34.584342 17455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:34.593658 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.593752 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.593735 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.593916 17455 server_base.cc:1034] running on GCE node
I20260619 11:10:34.594378 17455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:34.595013 17455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:34.596205 17455 hybrid_clock.cc:648] HybridClock initialized: now 1781867434596170 us; error 45 us; skew 500 ppm
I20260619 11:10:34.598052 17455 webserver.cc:460] Webserver started at http://127.7.90.66:39169/ using document root <none> and password file <none>
I20260619 11:10:34.598675 17455 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:34.598765 17455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:34.598994 17455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:34.601033 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "e9380b7b791c444d80f38617db47e770"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "a5e55a6dc03db1bd8bd736ab58a77bd9"
server_key_iv: "9e6632cc5b9f27380d4e5a2e4d6aacd2"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.601653 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "e9380b7b791c444d80f38617db47e770"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "a5e55a6dc03db1bd8bd736ab58a77bd9"
server_key_iv: "9e6632cc5b9f27380d4e5a2e4d6aacd2"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.605242 17455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:34.607779 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.608945 17455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:10:34.609086 17455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "e9380b7b791c444d80f38617db47e770"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "a5e55a6dc03db1bd8bd736ab58a77bd9"
server_key_iv: "9e6632cc5b9f27380d4e5a2e4d6aacd2"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.609230 17455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:34.613639 17455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:34.614202 17455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:34.614382 17455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:34.614902 17455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:34.615741 17455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:34.615813 17455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.615875 17455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:34.615929 17455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.624472 17455 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45389
I20260619 11:10:34.624493 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45389 every 8 connection(s)
I20260619 11:10:34.625244 17455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:34.630313 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17455
I20260619 11:10:34.630431 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:34.630767 7529 external_mini_cluster.cc:1428] Setting key 8fcf7047ea179b97a1fd1c81728d51f3
I20260619 11:10:34.631075 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.626938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:49399 (local address 127.7.90.126:39011)
0619 11:10:34.627081 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.627085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.628037 (+ 952us) server_negotiation.cc:407] Connection header received
0619 11:10:34.629321 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.629324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.629367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.629478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.630633 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:34.630636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:34.630638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:34.630650 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:34.630668 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:34.630676 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:34.630754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:34.630947 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 11:10:34.630982 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:10:34.631999 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.627312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39011 (local address 127.7.90.66:49399)
0619 11:10:34.627901 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:34.627936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:34.629112 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:34.629660 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:34.629666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:34.629744 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:34.630270 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:34.630272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:34.630336 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:34.630423 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:34.630764 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:34.630772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:34.630779 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:10:34.630946 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 11:10:34.631178 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260619 11:10:34.632726 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--builtin_ntp_servers=127.7.90.84:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 11:10:34.633040 17583 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39011
I20260619 11:10:34.633270 17583 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:34.633708 17583 heartbeater.cc:511] Master 127.7.90.126:39011 requested a full tablet report, sending...
I20260619 11:10:34.634719 17263 ts_manager.cc:194] Registered new tserver with Master: e9380b7b791c444d80f38617db47e770 (127.7.90.66:45389)
W20260619 11:10:34.741659 17586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:34.741900 17586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:34.741938 17586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:34.745956 17586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:34.746001 17586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:34.746076 17586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:34.750712 17586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:34.751695 17586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:34.752980 17586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:34.762061 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:34.762082 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.762151 17586 server_base.cc:1034] running on GCE node
W20260619 11:10:34.762075 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:34.762694 17586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:34.763235 17586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:34.764411 17586 hybrid_clock.cc:648] HybridClock initialized: now 1781867434764395 us; error 26 us; skew 500 ppm
I20260619 11:10:34.766052 17586 webserver.cc:460] Webserver started at http://127.7.90.67:46149/ using document root <none> and password file <none>
I20260619 11:10:34.766584 17586 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:34.766666 17586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:34.766883 17586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:34.768813 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "b77c077ddc054df69417af286780f2c1"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "f8e569cb1cfdac85c5f0932bcb839f72"
server_key_iv: "c4f0f9615f3e4b164f198b67d37322b4"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.769394 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "b77c077ddc054df69417af286780f2c1"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "f8e569cb1cfdac85c5f0932bcb839f72"
server_key_iv: "c4f0f9615f3e4b164f198b67d37322b4"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.773159 17586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:10:34.775650 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.776820 17586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:34.776952 17586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "b77c077ddc054df69417af286780f2c1"
format_stamp: "Formatted at 2026-06-19 11:10:34 on dist-test-slave-f1t5"
server_key: "f8e569cb1cfdac85c5f0932bcb839f72"
server_key_iv: "c4f0f9615f3e4b164f198b67d37322b4"
server_key_version: "encryptionkey@0"
I20260619 11:10:34.777069 17586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:34.781615 17586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:34.782188 17586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:34.782364 17586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:34.782876 17586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:34.783736 17586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:34.783805 17586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.783870 17586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:34.783915 17586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:34.792330 17586 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:41035
I20260619 11:10:34.792393 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:41035 every 8 connection(s)
I20260619 11:10:34.793197 17586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:34.799011 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.794981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:38951 (local address 127.7.90.126:39011)
0619 11:10:34.795118 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.795122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.796176 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 11:10:34.797461 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.797464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.797506 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.797615 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.798645 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:34.798648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:34.798650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:10:34.798661 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:34.798676 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:10:34.798684 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:34.798755 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:34.798870 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 11:10:34.798905 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:10:34.799366 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17586
I20260619 11:10:34.799449 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:34.799767 7529 external_mini_cluster.cc:1428] Setting key d2cf43e136d786afefdab901e1a9b558
I20260619 11:10:34.799947 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:34.795375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:39011 (local address 127.7.90.67:38951)
0619 11:10:34.796040 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:34.796076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:34.797254 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:34.797776 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:34.797783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:34.797866 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:34.798413 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:10:34.798415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:34.798450 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:10:34.798514 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:34.798757 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:34.798763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:34.798766 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:10:34.798889 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 11:10:34.799071 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260619 11:10:34.800951 17714 heartbeater.cc:347] Connected to a master server at 127.7.90.126:39011
I20260619 11:10:34.801182 17714 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:34.801638 17714 heartbeater.cc:511] Master 127.7.90.126:39011 requested a full tablet report, sending...
I20260619 11:10:34.802565 17264 ts_manager.cc:194] Registered new tserver with Master: b77c077ddc054df69417af286780f2c1 (127.7.90.67:41035)
I20260619 11:10:34.813544 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:10:34.816730 17303 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:10:34.815784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.7.90.126:39011)
0619 11:10:34.815938 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:34.815943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:34.816108 (+ 165us) server_negotiation.cc:407] Connection header received
0619 11:10:34.816255 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:34.816258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:34.816292 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:34.816378 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:34.816604 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41528: BlockingRecv error: recv got EOF from 127.0.0.1:41528 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:10:34.817091 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17324
I20260619 11:10:34.823303 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17455
I20260619 11:10:34.828781 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17586
I20260619 11:10:34.833810 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17233
2026-06-19T11:10:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:34 dist-test-slave-f1t5 krb5kdc[17724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:34 dist-test-slave-f1t5 krb5kdc[17724](info): set up 2 sockets
Jun 19 11:10:34 dist-test-slave-f1t5 krb5kdc[17724](info): commencing operation
krb5kdc: starting...
W20260619 11:10:35.866489 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:35 dist-test-slave-f1t5 krb5kdc[17724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:10:35.982530 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:34415
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:36.091698 17737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:36.092043 17737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:36.092136 17737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:36.096323 17737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:10:36.096429 17737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:36.096485 17737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:36.096555 17737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:10:36.096611 17737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:10:36.102309 17737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:36.103467 17737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:36.104521 17737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:36.113242 17743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.113253 17742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.113281 17737 server_base.cc:1034] running on GCE node
W20260619 11:10:36.113242 17745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.114110 17737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:36.115221 17737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:36.116407 17737 hybrid_clock.cc:648] HybridClock initialized: now 1781867436116393 us; error 28 us; skew 500 ppm
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:36.122049 17737 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:10:36.122910 17737 webserver.cc:460] Webserver started at http://127.7.90.126:42623/ using document root <none> and password file <none>
I20260619 11:10:36.123476 17737 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:36.123565 17737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:36.123792 17737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:36.125916 17737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "a240c061c1d046fa85e22369bd83cf05"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "62761aae859878ea76aba28ab2faadb3"
server_key_iv: "8469100f9f83f5a88ec85bd46e12e838"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.126475 17737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "a240c061c1d046fa85e22369bd83cf05"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "62761aae859878ea76aba28ab2faadb3"
server_key_iv: "8469100f9f83f5a88ec85bd46e12e838"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.130029 17737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:10:36.132370 17752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.133567 17737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:10:36.133719 17737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "a240c061c1d046fa85e22369bd83cf05"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "62761aae859878ea76aba28ab2faadb3"
server_key_iv: "8469100f9f83f5a88ec85bd46e12e838"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.133823 17737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:36.142720 17737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:36.145264 17737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:36.145465 17737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:36.151928 17737 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:34415
I20260619 11:10:36.152223 17804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:34415 every 8 connection(s)
I20260619 11:10:36.152859 17737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:10:36.155560 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:36.158416 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17737
I20260619 11:10:36.158584 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:10:36.158918 7529 external_mini_cluster.cc:1428] Setting key 485c3084afb252c05c8188a098d08799
I20260619 11:10:36.161432 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: Bootstrap starting.
I20260619 11:10:36.164052 17805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:36.164836 17805 log.cc:824] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:36.166993 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: No bootstrap required, opened a new log
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:36.170591 17805 raft_consensus.cc:348] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } }
I20260619 11:10:36.170837 17805 raft_consensus.cc:374] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:36.170949 17805 raft_consensus.cc:729] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a240c061c1d046fa85e22369bd83cf05, State: Initialized, Role: FOLLOWER
I20260619 11:10:36.171452 17805 consensus_queue.cc:260] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } }
I20260619 11:10:36.171604 17805 raft_consensus.cc:388] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:10:36.171687 17805 raft_consensus.cc:482] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:10:36.171787 17805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:36.172796 17805 raft_consensus.cc:504] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } }
I20260619 11:10:36.173156 17805 leader_election.cc:302] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a240c061c1d046fa85e22369bd83cf05; no voters:
I20260619 11:10:36.173251 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.160323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.7.90.126:34415)
0619 11:10:36.160713 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.160731 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.160773 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:10:36.161593 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.161615 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.162022 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.162373 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:36.163017 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.164457 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.165325 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.165900 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.166017 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.168706 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:36.168711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:36.168723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:36.168748 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:36.170449 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.170901 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.170907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.170919 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.171019 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.171332 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.171335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.171336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.171745 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:36.171982 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.172203 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.172479 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260619 11:10:36.173466 17805 leader_election.cc:288] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:10:36.173571 17810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:36.173792 17810 raft_consensus.cc:686] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [term 1 LEADER]: Becoming Leader. State: Replica: a240c061c1d046fa85e22369bd83cf05, State: Running, Role: LEADER
I20260619 11:10:36.174140 17810 consensus_queue.cc:237] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } }
I20260619 11:10:36.174722 17805 sys_catalog.cc:564] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:10:36.175258 17810 sys_catalog.cc:455] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a240c061c1d046fa85e22369bd83cf05. Latest consensus state: current_term: 1 leader_uuid: "a240c061c1d046fa85e22369bd83cf05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } } }
I20260619 11:10:36.175402 17810 sys_catalog.cc:458] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:36.175827 17817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:10:36.176433 17811 sys_catalog.cc:455] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a240c061c1d046fa85e22369bd83cf05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a240c061c1d046fa85e22369bd83cf05" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 34415 } } }
I20260619 11:10:36.176631 17811 sys_catalog.cc:458] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05 [sys.catalog]: This master's current role is: LEADER
I20260619 11:10:36.178905 17817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:10:36.184392 17817 catalog_manager.cc:1347] Generated new cluster ID: 0d2d8377d0984c0b8c664065a1524960
I20260619 11:10:36.184468 17817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:10:36.193866 17817 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:10:36.194752 17817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:10:36.204217 17817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: Generated new TSK 0
I20260619 11:10:36.204900 17817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:10:36.231904 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:36.341604 17831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:36.341888 17831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:36.341930 17831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:36.345783 17831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:36.345829 17831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:36.345883 17831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:10:36.350903 17831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:36.351819 17831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:36.352736 17831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:36.362071 17837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.362087 17836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.362099 17839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.362390 17831 server_base.cc:1034] running on GCE node
I20260619 11:10:36.362792 17831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:36.363333 17831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:36.364491 17831 hybrid_clock.cc:648] HybridClock initialized: now 1781867436364475 us; error 27 us; skew 500 ppm
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:36.367161 17831 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:10:36.367980 17831 webserver.cc:460] Webserver started at http://127.7.90.65:45097/ using document root <none> and password file <none>
I20260619 11:10:36.368467 17831 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:36.368520 17831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:36.368673 17831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:36.370602 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "82027be64dee405a81d57a023966570b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "427d68aa40c9db74730a315a89cfbd5e"
server_key_iv: "ee864a33e1b0f9c344a51790a7128c9a"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.371094 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "82027be64dee405a81d57a023966570b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "427d68aa40c9db74730a315a89cfbd5e"
server_key_iv: "ee864a33e1b0f9c344a51790a7128c9a"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.374670 17831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:10:36.377131 17846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.378368 17831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:10:36.378499 17831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "82027be64dee405a81d57a023966570b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "427d68aa40c9db74730a315a89cfbd5e"
server_key_iv: "ee864a33e1b0f9c344a51790a7128c9a"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.378604 17831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:36.391160 17831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:36.394068 17831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:36.394263 17831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:36.394834 17831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:36.395638 17831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:36.395705 17831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.395776 17831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:36.395826 17831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.404844 17831 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36133
I20260619 11:10:36.404868 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36133 every 8 connection(s)
I20260619 11:10:36.405761 17831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:10:36.407702 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17831
I20260619 11:10:36.407831 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:10:36.408111 7529 external_mini_cluster.cc:1428] Setting key 685742806ae3f15e59201b70a3e59774
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:10:36.421223 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.407585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:42227 (local address 127.7.90.126:34415)
0619 11:10:36.407743 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.407748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.408637 (+ 889us) server_negotiation.cc:407] Connection header received
0619 11:10:36.410100 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.410106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.410168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.410262 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:36.411662 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.412347 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.413601 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.414098 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.414286 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.417119 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:36.417122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:36.417124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:36.417147 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:36.418698 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.419306 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.419309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.419310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.419382 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.419831 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.419833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.419835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.420006 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:36.420099 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.420817 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.420960 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:10:36.421922 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.408014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:34415 (local address 127.7.90.65:42227)
0619 11:10:36.408474 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.408536 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.409899 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.410413 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.410420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.410807 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:36.411501 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.411520 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.412473 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.412476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.413445 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.413454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.414312 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.414315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.414457 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.415066 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:36.415082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:36.416933 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:36.418928 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.418934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.418946 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.419209 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.419506 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.419512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.419514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.419720 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.420181 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:36.420189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:36.420523 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.420737 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.420998 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260619 11:10:36.423303 17960 heartbeater.cc:347] Connected to a master server at 127.7.90.126:34415
I20260619 11:10:36.423563 17960 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:36.424037 17960 heartbeater.cc:511] Master 127.7.90.126:34415 requested a full tablet report, sending...
I20260619 11:10:36.425804 17769 ts_manager.cc:194] Registered new tserver with Master: 82027be64dee405a81d57a023966570b (127.7.90.65:36133)
I20260619 11:10:36.427482 17769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:42227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:10:36.435559 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:36.547508 17965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:36.547785 17965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:36.547832 17965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:36.552088 17965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:36.552155 17965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:36.552222 17965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:10:36.557603 17965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:36.558637 17965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:36.559592 17965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:10:36.569314 17965 server_base.cc:1034] running on GCE node
W20260619 11:10:36.569347 17971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.569284 17970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.569597 17973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.569978 17965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:36.570643 17965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:36.571805 17965 hybrid_clock.cc:648] HybridClock initialized: now 1781867436571786 us; error 67 us; skew 500 ppm
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:36.574826 17965 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:10:36.575906 17965 webserver.cc:460] Webserver started at http://127.7.90.66:43241/ using document root <none> and password file <none>
I20260619 11:10:36.576524 17965 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:36.576623 17965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:36.576844 17965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:36.578997 17965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "20dce1860b794f9c93f3e449a8c7dd0c"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "233f321b03f43d1a0a40ba294292a442"
server_key_iv: "a5b9cff1bba7db7d65555fc32e308e0b"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.579566 17965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "20dce1860b794f9c93f3e449a8c7dd0c"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "233f321b03f43d1a0a40ba294292a442"
server_key_iv: "a5b9cff1bba7db7d65555fc32e308e0b"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.583348 17965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 11:10:36.585798 17980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.587121 17965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:10:36.587239 17965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "20dce1860b794f9c93f3e449a8c7dd0c"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "233f321b03f43d1a0a40ba294292a442"
server_key_iv: "a5b9cff1bba7db7d65555fc32e308e0b"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.587314 17965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:36.607708 17965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:36.610761 17965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:36.610935 17965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:36.611505 17965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:36.612388 17965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:36.612440 17965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.612514 17965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:36.612555 17965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.621807 17965 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:34283
I20260619 11:10:36.621834 18093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:34283 every 8 connection(s)
I20260619 11:10:36.622737 17965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:10:36.632102 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 17965
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:36.632195 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:10:36.632575 7529 external_mini_cluster.cc:1428] Setting key 0915183129de1730206a900368b88e68
I20260619 11:10:36.636979 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.624511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:36957 (local address 127.7.90.126:34415)
0619 11:10:36.624644 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.624656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.625397 (+ 741us) server_negotiation.cc:407] Connection header received
0619 11:10:36.626850 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.626854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.626951 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.627050 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:36.628202 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.628725 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.629957 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.630179 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.630296 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.632873 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:36.632878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:36.632880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:36.632911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:36.634718 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.635232 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.635238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.635243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.635339 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.635660 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.635666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.635671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.635981 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:36.636103 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.636744 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.636847 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:10:36.637971 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.624809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:34415 (local address 127.7.90.66:36957)
0619 11:10:36.625229 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.625276 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.626630 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.627231 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.627238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.627587 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:36.628050 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.628063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.628851 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.628854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.629816 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.629826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.630330 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.630334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.630511 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.631032 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:36.631049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:36.632656 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:36.634857 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.634861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.634873 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.635121 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.635433 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.635435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.635436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.635570 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.636112 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:36.636120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:36.636442 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.636659 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.636998 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260619 11:10:36.639175 18094 heartbeater.cc:347] Connected to a master server at 127.7.90.126:34415
I20260619 11:10:36.639418 18094 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:36.639892 18094 heartbeater.cc:511] Master 127.7.90.126:34415 requested a full tablet report, sending...
I20260619 11:10:36.640892 17769 ts_manager.cc:194] Registered new tserver with Master: 20dce1860b794f9c93f3e449a8c7dd0c (127.7.90.66:34283)
I20260619 11:10:36.641425 17769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:36957
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:10:36.658464 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:10:36.768179 18099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:10:36.768533 18099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:10:36.768586 18099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:10:36.772372 18099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:10:36.772421 18099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:10:36.772475 18099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:10:36.777558 18099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:10:36.778467 18099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:10:36.779381 18099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:10:36.788710 18105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:10:36.788745 18104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.788766 18099 server_base.cc:1034] running on GCE node
W20260619 11:10:36.788730 18107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:10:36.789304 18099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:10:36.789917 18099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:10:36.791067 18099 hybrid_clock.cc:648] HybridClock initialized: now 1781867436791037 us; error 43 us; skew 500 ppm
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:10:36.793817 18099 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:10:36.794574 18099 webserver.cc:460] Webserver started at http://127.7.90.67:40785/ using document root <none> and password file <none>
I20260619 11:10:36.795063 18099 fs_manager.cc:359] Metadata directory not provided
I20260619 11:10:36.795126 18099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:10:36.795290 18099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:10:36.797166 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "bc6e1115e546473e924b33baf2e74d5b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "52d0a86a5a307c9f97e55506a9e3e723"
server_key_iv: "2ce212c2b5fe34678c27029d5d41a384"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.797732 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "bc6e1115e546473e924b33baf2e74d5b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "52d0a86a5a307c9f97e55506a9e3e723"
server_key_iv: "2ce212c2b5fe34678c27029d5d41a384"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.801256 18099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:10:36.803508 18114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.804697 18099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:10:36.804824 18099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "bc6e1115e546473e924b33baf2e74d5b"
format_stamp: "Formatted at 2026-06-19 11:10:36 on dist-test-slave-f1t5"
server_key: "52d0a86a5a307c9f97e55506a9e3e723"
server_key_iv: "2ce212c2b5fe34678c27029d5d41a384"
server_key_version: "encryptionkey@0"
I20260619 11:10:36.804927 18099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:10:36.816138 18099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:10:36.818854 18099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:10:36.819054 18099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:10:36.819599 18099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:10:36.820369 18099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:10:36.820437 18099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.820505 18099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:10:36.820549 18099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:10:36.829124 18099 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:42685
I20260619 11:10:36.829139 18227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:42685 every 8 connection(s)
I20260619 11:10:36.830020 18099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:10:36.834098 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 18099
I20260619 11:10:36.834223 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:10:36.834498 7529 external_mini_cluster.cc:1428] Setting key 78fa8240701a56b5bdcf7f2c83c9cd09
Jun 19 11:10:36 dist-test-slave-f1t5 krb5kdc[17724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867436, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:10:36.843091 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.831756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:58179 (local address 127.7.90.126:34415)
0619 11:10:36.831917 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.831922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.832719 (+ 797us) server_negotiation.cc:407] Connection header received
0619 11:10:36.834028 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.834033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.834104 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.834205 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:36.835611 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.836202 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.837494 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.837742 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.837819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.839991 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:36.839994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:36.839996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:36.840016 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:36.841156 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.841627 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.841632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.841633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.841713 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.842056 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.842061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.842062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.842231 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:36.842358 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.842811 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.842928 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:10:36.843822 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.832122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:34415 (local address 127.7.90.67:58179)
0619 11:10:36.832565 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.832609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.833830 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.834382 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.834388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.834801 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:10:36.835481 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.835497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.836325 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.836328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.837378 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.837387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.837858 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.837860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.838025 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.838536 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:10:36.838551 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:10:36.839856 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:10:36.841250 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.841254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.841263 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.841488 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.841828 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:10:36.841834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:10:36.841836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:10:36.841961 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:10:36.842365 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:10:36.842373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:10:36.842588 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.842772 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.843070 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
I20260619 11:10:36.844815 18228 heartbeater.cc:347] Connected to a master server at 127.7.90.126:34415
I20260619 11:10:36.845045 18228 heartbeater.cc:464] Registering TS with master...
I20260619 11:10:36.845504 18228 heartbeater.cc:511] Master 127.7.90.126:34415 requested a full tablet report, sending...
I20260619 11:10:36.846530 17769 ts_manager.cc:194] Registered new tserver with Master: bc6e1115e546473e924b33baf2e74d5b (127.7.90.67:42685)
I20260619 11:10:36.847157 17769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:58179
I20260619 11:10:36.848083 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:10:36.856458 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.849890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.7.90.126:34415)
0619 11:10:36.849997 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.850000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.850182 (+ 182us) server_negotiation.cc:407] Connection header received
0619 11:10:36.850274 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.850276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.850307 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.850368 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:10:36.850945 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.851399 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.852155 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.852348 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.852418 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.853673 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:10:36.853676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:10:36.853677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:10:36.853698 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:10:36.855193 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.855539 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.855541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.855543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.855619 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:10:36.855832 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:10:36.855838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:10:36.855840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:10:36.856016 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:10:36.856114 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.856272 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.856352 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:10:36.859263 17769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:10:36.862318 17769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:10:36.877249 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.870715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:42685 (local address 127.0.0.1:47434)
0619 11:10:36.871517 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.871564 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.871686 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.872357 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.872363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.872385 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:36.872528 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.872534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.873997 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.874000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.875498 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.875507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.876654 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.876657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.876813 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.876876 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.876970 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.877102 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260619 11:10:36.877307 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.870977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.7.90.67:42685)
0619 11:10:36.871832 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.871841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.871861 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:10:36.871920 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.871924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.872147 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.872363 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:36.872651 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.873853 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.875603 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.876506 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.876859 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.877008 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.877071 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.877161 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":151,"threads_started":1}
I20260619 11:10:36.878016 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.870779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36133 (local address 127.0.0.1:46012)
0619 11:10:36.871902 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.871917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.871992 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.872646 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.872649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.872666 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:36.872760 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.872768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.874025 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.874028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.876348 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.876355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.877628 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.877632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.877711 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.877730 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.877807 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.877877 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":41,"thread_start_us":89,"threads_started":1}
I20260619 11:10:36.878125 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.871374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.7.90.65:36133)
0619 11:10:36.872242 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.872254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.872275 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:10:36.872349 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.872354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.872507 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.872658 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:36.872887 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.873872 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.876467 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.877304 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.877417 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.877556 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.877871 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.877964 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":121,"threads_started":1}
I20260619 11:10:36.879050 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.871414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:34283 (local address 127.0.0.1:49848)
0619 11:10:36.872165 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:36.872178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:36.872273 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:36.873037 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:36.873040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:36.873059 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:36.873131 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.873137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.874500 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.874503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.875678 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:36.875686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.876775 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.876778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:36.878788 (+ 2010us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.878810 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:10:36.878887 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:10:36.878946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260619 11:10:36.879470 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:36.871809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.7.90.66:34283)
0619 11:10:36.872130 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:36.872141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:36.872274 (+ 133us) server_negotiation.cc:407] Connection header received
0619 11:10:36.872720 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:36.872726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:36.872900 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:36.873052 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:36.873225 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.874355 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.875781 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:36.876513 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:36.876631 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:36.876820 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:36.879277 (+ 2457us) server_negotiation.cc:299] Negotiation successful
0619 11:10:36.879369 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260619 11:10:36.880841 18162 tablet_service.cc:1467] Processing CreateTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 (DEFAULT_TABLE table=test-table [id=6b1a7006c6d244feb736e18f795b746d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:36.880841 17894 tablet_service.cc:1467] Processing CreateTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 (DEFAULT_TABLE table=test-table [id=6b1a7006c6d244feb736e18f795b746d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:36.881978 18162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b93d7e5dd84660a99a28e804c8e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:36.881976 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b93d7e5dd84660a99a28e804c8e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:36.881901 18028 tablet_service.cc:1467] Processing CreateTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 (DEFAULT_TABLE table=test-table [id=6b1a7006c6d244feb736e18f795b746d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:10:36.882833 18028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b93d7e5dd84660a99a28e804c8e882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:10:36.888419 18245 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Bootstrap starting.
I20260619 11:10:36.889568 18246 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Bootstrap starting.
I20260619 11:10:36.889912 18247 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Bootstrap starting.
I20260619 11:10:36.891186 18245 tablet_bootstrap.cc:654] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:36.891448 18246 tablet_bootstrap.cc:654] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:36.891913 18247 tablet_bootstrap.cc:654] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Neither blocks nor log segments found. Creating new log.
I20260619 11:10:36.892102 18245 log.cc:824] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:36.892683 18246 log.cc:824] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:36.893199 18247 log.cc:824] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:10:36.894219 18245 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: No bootstrap required, opened a new log
I20260619 11:10:36.894402 18245 ts_tablet_manager.cc:1397] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:10:36.894517 18246 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: No bootstrap required, opened a new log
I20260619 11:10:36.894713 18246 ts_tablet_manager.cc:1397] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:10:36.895004 18247 tablet_bootstrap.cc:492] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: No bootstrap required, opened a new log
I20260619 11:10:36.895174 18247 ts_tablet_manager.cc:1397] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:10:36.897420 18246 raft_consensus.cc:348] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.897789 18246 raft_consensus.cc:374] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:36.897863 18246 raft_consensus.cc:729] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82027be64dee405a81d57a023966570b, State: Initialized, Role: FOLLOWER
I20260619 11:10:36.898298 18246 consensus_queue.cc:260] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.898260 18245 raft_consensus.cc:348] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.898514 18245 raft_consensus.cc:374] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:36.898636 18245 raft_consensus.cc:729] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20dce1860b794f9c93f3e449a8c7dd0c, State: Initialized, Role: FOLLOWER
I20260619 11:10:36.898762 18247 raft_consensus.cc:348] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.899004 18247 raft_consensus.cc:374] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:10:36.899056 18247 raft_consensus.cc:729] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6e1115e546473e924b33baf2e74d5b, State: Initialized, Role: FOLLOWER
I20260619 11:10:36.899104 18245 consensus_queue.cc:260] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.899483 18246 ts_tablet_manager.cc:1428] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:10:36.899494 18247 consensus_queue.cc:260] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:36.899750 18094 heartbeater.cc:503] Master 127.7.90.126:34415 was elected leader, sending a full tablet report...
I20260619 11:10:36.899811 17960 heartbeater.cc:503] Master 127.7.90.126:34415 was elected leader, sending a full tablet report...
I20260619 11:10:36.899982 18245 ts_tablet_manager.cc:1428] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 11:10:36.900871 18228 heartbeater.cc:503] Master 127.7.90.126:34415 was elected leader, sending a full tablet report...
I20260619 11:10:36.901160 18247 ts_tablet_manager.cc:1428] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260619 11:10:36.906854 17961 tablet.cc:2369] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:37.044656 18252 raft_consensus.cc:482] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:10:37.044889 18252 raft_consensus.cc:504] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:37.046059 18252 leader_election.cc:288] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc6e1115e546473e924b33baf2e74d5b (127.7.90.67:42685), 82027be64dee405a81d57a023966570b (127.7.90.65:36133)
I20260619 11:10:37.049688 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.046340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:42685 (local address 127.7.90.66:50857)
0619 11:10:37.046560 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:37.046577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:37.046677 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:37.046935 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:37.046937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:37.046952 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:37.047067 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:37.047072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.047771 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.047776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:37.048544 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:37.048552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.049221 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.049225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:37.049341 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:37.049357 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:10:37.049466 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:10:37.049557 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 11:10:37.049710 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.046423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:50857 (local address 127.7.90.67:42685)
0619 11:10:37.046547 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:37.046554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:37.046747 (+ 193us) server_negotiation.cc:407] Connection header received
0619 11:10:37.046796 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:37.046798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:37.046838 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:37.046923 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:37.047179 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.047659 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.048714 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.049103 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.049218 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:37.049289 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:37.049492 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 11:10:37.049588 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:10:37.050122 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.046504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:38901 (local address 127.7.90.65:36133)
0619 11:10:37.046632 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:37.046637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:37.046849 (+ 212us) server_negotiation.cc:407] Connection header received
0619 11:10:37.046997 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:37.047000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:37.047049 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:37.047138 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:10:37.047379 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.047841 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.049009 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.049489 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.049643 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:37.049714 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:37.049941 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:10:37.050013 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:10:37.050130 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.046340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36133 (local address 127.7.90.66:38901)
0619 11:10:37.046751 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:37.046761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:10:37.046889 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:10:37.047145 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:10:37.047147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:10:37.047161 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:10:37.047277 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:37.047282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.047943 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.047949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:37.048894 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:10:37.048903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.049643 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.049646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:10:37.049815 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:37.049849 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:10:37.049929 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:10:37.050000 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":31,"thread_start_us":127,"threads_started":1}
I20260619 11:10:37.050400 18182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b93d7e5dd84660a99a28e804c8e882" candidate_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6e1115e546473e924b33baf2e74d5b" is_pre_election: true
I20260619 11:10:37.050642 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b93d7e5dd84660a99a28e804c8e882" candidate_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82027be64dee405a81d57a023966570b" is_pre_election: true
I20260619 11:10:37.050700 18182 raft_consensus.cc:2454] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dce1860b794f9c93f3e449a8c7dd0c in term 0.
I20260619 11:10:37.050917 17914 raft_consensus.cc:2454] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dce1860b794f9c93f3e449a8c7dd0c in term 0.
I20260619 11:10:37.051200 17983 leader_election.cc:302] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20dce1860b794f9c93f3e449a8c7dd0c, bc6e1115e546473e924b33baf2e74d5b; no voters:
I20260619 11:10:37.051497 18252 raft_consensus.cc:2781] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:10:37.051573 18252 raft_consensus.cc:482] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:10:37.051630 18252 raft_consensus.cc:3037] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:37.052613 18252 raft_consensus.cc:504] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:37.052954 18252 leader_election.cc:288] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [CANDIDATE]: Term 1 election: Requested vote from peers bc6e1115e546473e924b33baf2e74d5b (127.7.90.67:42685), 82027be64dee405a81d57a023966570b (127.7.90.65:36133)
I20260619 11:10:37.053237 18182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b93d7e5dd84660a99a28e804c8e882" candidate_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6e1115e546473e924b33baf2e74d5b"
I20260619 11:10:37.053256 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b93d7e5dd84660a99a28e804c8e882" candidate_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82027be64dee405a81d57a023966570b"
I20260619 11:10:37.053356 17914 raft_consensus.cc:3037] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:37.053372 18182 raft_consensus.cc:3037] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:10:37.054363 17914 raft_consensus.cc:2454] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dce1860b794f9c93f3e449a8c7dd0c in term 1.
I20260619 11:10:37.054451 18182 raft_consensus.cc:2454] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dce1860b794f9c93f3e449a8c7dd0c in term 1.
I20260619 11:10:37.054651 17984 leader_election.cc:302] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20dce1860b794f9c93f3e449a8c7dd0c, 82027be64dee405a81d57a023966570b; no voters:
I20260619 11:10:37.054818 18252 raft_consensus.cc:2781] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:10:37.055087 18252 raft_consensus.cc:686] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 1 LEADER]: Becoming Leader. State: Replica: 20dce1860b794f9c93f3e449a8c7dd0c, State: Running, Role: LEADER
I20260619 11:10:37.055413 18252 consensus_queue.cc:237] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } }
I20260619 11:10:37.058357 17769 catalog_manager.cc:5495] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 20dce1860b794f9c93f3e449a8c7dd0c (127.7.90.66). New cstate: current_term: 1 leader_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20dce1860b794f9c93f3e449a8c7dd0c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34283 } health_report { overall_health: HEALTHY } } }
W20260619 11:10:37.080933 18229 tablet.cc:2369] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:10:37.108664 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.105568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.7.90.66:34283)
0619 11:10:37.105702 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:10:37.105706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:10:37.105719 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:10:37.105787 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:10:37.105790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:10:37.105845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:10:37.105934 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:10:37.106143 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.106636 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.107405 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:10:37.107605 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:10:37.107674 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:10:37.107946 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:10:37.108342 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:10:37.108432 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:10:37.108491 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 11:10:37.108535 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:10:37.114257 17914 raft_consensus.cc:1261] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 1 FOLLOWER]: Refusing update from remote peer 20dce1860b794f9c93f3e449a8c7dd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:37.114257 18182 raft_consensus.cc:1261] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 1 FOLLOWER]: Refusing update from remote peer 20dce1860b794f9c93f3e449a8c7dd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:10:37.114919 18252 consensus_queue.cc:1035] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6e1115e546473e924b33baf2e74d5b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:37.115135 18255 consensus_queue.cc:1035] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82027be64dee405a81d57a023966570b" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:10:37.129386 17769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:37.129575 17769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:10:37.131760 17769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a240c061c1d046fa85e22369bd83cf05: Sending DeleteTablet for 3 replicas of tablet 70b93d7e5dd84660a99a28e804c8e882
I20260619 11:10:37.132477 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:37 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:46012
I20260619 11:10:37.132479 18162 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:37 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47434
I20260619 11:10:37.132951 18268 tablet_replica.cc:330] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: stopping tablet replica
I20260619 11:10:37.132939 18028 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b93d7e5dd84660a99a28e804c8e882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:10:37 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:49848
I20260619 11:10:37.133251 18268 raft_consensus.cc:2229] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:37.133427 18270 tablet_replica.cc:330] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: stopping tablet replica
I20260619 11:10:37.133562 18268 raft_consensus.cc:2258] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:37.133908 18269 tablet_replica.cc:330] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b: stopping tablet replica
I20260619 11:10:37.134078 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17831
I20260619 11:10:37.134073 18270 raft_consensus.cc:2229] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:10:37.134218 18269 raft_consensus.cc:2229] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:10:37.134471 18269 raft_consensus.cc:2258] T 70b93d7e5dd84660a99a28e804c8e882 P 82027be64dee405a81d57a023966570b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:37.134946 18268 ts_tablet_manager.cc:1905] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:37.134388 18270 raft_consensus.cc:2258] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:10:37.136525 18270 ts_tablet_manager.cc:1905] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:10:37.137300 18268 ts_tablet_manager.cc:1918] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:37.137382 18268 log.cc:1196] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/70b93d7e5dd84660a99a28e804c8e882
I20260619 11:10:37.137760 18268 ts_tablet_manager.cc:1939] T 70b93d7e5dd84660a99a28e804c8e882 P bc6e1115e546473e924b33baf2e74d5b: Deleting consensus metadata
I20260619 11:10:37.139946 18270 ts_tablet_manager.cc:1918] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:10:37.140117 18270 log.cc:1196] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/70b93d7e5dd84660a99a28e804c8e882
I20260619 11:10:37.140477 18270 ts_tablet_manager.cc:1939] T 70b93d7e5dd84660a99a28e804c8e882 P 20dce1860b794f9c93f3e449a8c7dd0c: Deleting consensus metadata
I20260619 11:10:37.141793 17755 catalog_manager.cc:4841] TS 20dce1860b794f9c93f3e449a8c7dd0c (127.7.90.66:34283): tablet 70b93d7e5dd84660a99a28e804c8e882 (table test-table [id=6b1a7006c6d244feb736e18f795b746d]) successfully deleted
W20260619 11:10:37.141916 17756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:36133 (error 108)
I20260619 11:10:37.141968 17755 catalog_manager.cc:4841] TS bc6e1115e546473e924b33baf2e74d5b (127.7.90.67:42685): tablet 70b93d7e5dd84660a99a28e804c8e882 (table test-table [id=6b1a7006c6d244feb736e18f795b746d]) successfully deleted
I20260619 11:10:37.142144 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17965
I20260619 11:10:37.146360 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:10:37.145977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36133 (local address 127.0.0.1:46018)
0619 11:10:37.146174 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 11:10:37.146258 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 11:10:37.146533 17756 catalog_manager.cc:4584] TS 82027be64dee405a81d57a023966570b (127.7.90.65:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70b93d7e5dd84660a99a28e804c8e882: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36133: connect: Connection refused (error 111)
I20260619 11:10:37.148041 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 18099
I20260619 11:10:37.153347 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 17737
2026-06-19T11:10:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:10:37 dist-test-slave-f1t5 krb5kdc[18274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:10:37 dist-test-slave-f1t5 krb5kdc[18274](info): set up 2 sockets
Jun 19 11:10:37 dist-test-slave-f1t5 krb5kdc[18274](info): commencing operation
krb5kdc: starting...
W20260619 11:10:38.190470 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:10:38 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:10:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangeradmin_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangeradmin_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangerlookup_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangerlookup_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
I20260619 11:10:38.346449 7529 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7DZYiV/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/postgres -l logfile start
2026-06-19 11:10:40.651 UTC [18304] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:10:40.651 UTC [18304] LOG: listening on IPv4 address "127.7.90.84", port 59205
2026-06-19 11:10:40.654 UTC [18304] LOG: listening on Unix socket "/tmp/.s.PGSQL.59205"
2026-06-19 11:10:40.688 UTC [18307] LOG: database system was shut down at 2026-06-19 11:10:39 UTC
2026-06-19 11:10:40.692 UTC [18304] LOG: database system is ready to accept connections
I20260619 11:10:41.642161 7529 mini_postgres.cc:96] Postgres bound to 59205
2026-06-19 11:10:41.648 UTC [18315] FATAL: database "slave" does not exist
127.7.90.84:59205 - accepting connections
I20260619 11:10:41.649236 7529 mini_ranger.cc:161] Starting Ranger...
I20260619 11:10:41.663682 7529 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:10:42.093245 7529 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:10:42.093344 7529 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin
2026-06-19 11:10:42,344 [I] DB FLAVOR :POSTGRES
2026-06-19 11:10:42,344 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:10:42,344 [I] Checking connection..
2026-06-19 11:10:42,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:10:42,588 [I] Checking connection passed.
2026-06-19 11:10:42,588 [I] --------- Verifying version history table ---------
2026-06-19 11:10:42,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:10:42,857 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:10:42,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:10:43,101 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:10:43,102 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:10:43,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:10:43.318 UTC [18379] WARNING: there is no transaction in progress
2026-06-19 11:10:43,323 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:10:43,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:10:43,573 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:10:43,573 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:10:43,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:10:43,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:10:44,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f1t5.c.gcp-upstream.internal','N') ;"
2026-06-19 11:10:44,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:10:44,532 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:10:44,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:10:44,768 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:10:44,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:10:44,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:10:45,213 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:10:45,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:10:46.106 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.121 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.136 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.149 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.162 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.177 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.226 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.232 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.245 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.254 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.263 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.276 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.285 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.291 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.297 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46.303 UTC [18487] WARNING: there is no transaction in progress
2026-06-19 11:10:46,882 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:10:46,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:10:47,110 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:10:47,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:10:47,360 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:10:47,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:10:47,616 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:10:47,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:10:47,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:10:48,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f1t5.c.gcp-upstream.internal';"
2026-06-19 11:10:48,293 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:10:48,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:10:48,524 [I] DB_PATCHES have already been applied
I20260619 11:10:48.529850 7529 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7DZYiV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:10:48.529912 7529 mini_ranger.cc:193] Using host: 127.7.90.84
I20260619 11:10:48.533138 7529 mini_ranger.cc:227] Ranger admin URL: http://127.7.90.84:53679
Jun 19, 2026 11:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:10:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:10:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:10:49.611805 7529 mini_ranger.cc:160] Time spent starting Ranger: real 7.963s user 0.000s sys 0.010s
Jun 19, 2026 11:11:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:11:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 11:11:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:11:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:11:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:11:09.415004 7529 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangerkms_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/rangerkms_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.90.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.90.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:11:09 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:11:09.507136 7529 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:11:09.517129 7529 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:11:10.258769 7529 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:11:10.258867 7529 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms
I20260619 11:11:10.258991 7529 mini_ranger_kms.cc:227] Using postgres at 127.7.90.84:59205
2026-06-19 11:11:10,450 [I] DB FLAVOR :POSTGRES
2026-06-19 11:11:10,451 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:11:10,451 [I] Checking connection
2026-06-19 11:11:10,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:11:10,708 [I] connection success
2026-06-19 11:11:10,708 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:11:10,708 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:11:10,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:11:10,943 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:11:10,943 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:11:10,944 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:11:10,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:59205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:11:11,199 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:11:11.605242 7529 mini_ranger_kms.cc:310] Created kms service
I20260619 11:11:11.764591 7529 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:11:11.806738 7529 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:11:11 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:12.191860 7529 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:11:12.192015 7529 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms:/tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:11:12.192037 7529 mini_ranger_kms.cc:240] Using host: 127.7.90.84
I20260619 11:11:12.196122 7529 mini_ranger_kms.cc:276] Ranger KMS PID: 18667
I20260619 11:11:12.196240 7529 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.90.84:56611
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:11:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 11:11:14.421612 7529 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.914s user 0.004s sys 0.172s
I20260619 11:11:14.421725 7529 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:11:14.421856 7529 mini_ranger_kms.cc:401] 127.7.90.84:56611/kms/v1/keys
Jun 19 11:11:14 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 11:11:15 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
Jun 19 11:11:15 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
Jun 19 11:11:17 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:11:17.618875 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46629
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:46629
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:56611/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:11:17.727211 18718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:11:17.727487 18718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:11:17.727556 18718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:11:17.731320 18718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:11:17.731388 18718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:11:17.731422 18718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:11:17.731473 18718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:11:17.731506 18718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:11:17.736547 18718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.90.84:56611/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:46629
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:11:17.737571 18718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:11:17.738526 18718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:11:17.746687 18726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:17.746694 18724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:17.746702 18723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:11:17.746999 18718 server_base.cc:1034] running on GCE node
I20260619 11:11:17.747543 18718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:11:17.748548 18718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:11:17.749693 18718 hybrid_clock.cc:648] HybridClock initialized: now 1781867477749669 us; error 37 us; skew 500 ppm
Jun 19 11:11:17 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867477, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:11:17.752166 18718 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:11:17.752969 18718 webserver.cc:460] Webserver started at http://127.7.90.126:41599/ using document root <none> and password file <none>
I20260619 11:11:17.753439 18718 fs_manager.cc:359] Metadata directory not provided
I20260619 11:11:17.753489 18718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:11:17.753738 18718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:11:17 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867477, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:17.805658 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "e36b65112d4e40fcb9f233ee47b47949"
format_stamp: "Formatted at 2026-06-19 11:11:17 on dist-test-slave-f1t5"
server_key: "092dc7d04cfc3997f9b6469a693bca66"
server_key_iv: "a38d809d1ff57585c02b043046b8404a"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:17.806358 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "e36b65112d4e40fcb9f233ee47b47949"
format_stamp: "Formatted at 2026-06-19 11:11:17 on dist-test-slave-f1t5"
server_key: "092dc7d04cfc3997f9b6469a693bca66"
server_key_iv: "a38d809d1ff57585c02b043046b8404a"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:17.811028 18718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:11:17.825251 18733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:17.826607 18718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:11:17.826742 18718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "e36b65112d4e40fcb9f233ee47b47949"
format_stamp: "Formatted at 2026-06-19 11:11:17 on dist-test-slave-f1t5"
server_key: "092dc7d04cfc3997f9b6469a693bca66"
server_key_iv: "a38d809d1ff57585c02b043046b8404a"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:17.826870 18718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:11:17.846572 18718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:11:17.850080 18718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:11:17.850286 18718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:11:17.858295 18718 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:46629
I20260619 11:11:17.858319 18785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:46629 every 8 connection(s)
I20260619 11:11:17.859246 18718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:11:17.861900 18786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:11:17.865933 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 18718
I20260619 11:11:17.866080 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:11:17.866358 7529 external_mini_cluster.cc:1428] Setting key 2307edfa66d613bdd39c6cb04311e04c
I20260619 11:11:17.867640 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: Bootstrap starting.
I20260619 11:11:17.869882 18786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: Neither blocks nor log segments found. Creating new log.
I20260619 11:11:17.870594 18786 log.cc:824] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: Log is configured to *not* fsync() on all Append() calls
I20260619 11:11:17.872337 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: No bootstrap required, opened a new log
Jun 19 11:11:17 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:11:17.876288 18786 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } }
I20260619 11:11:17.876503 18786 raft_consensus.cc:374] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:11:17.876595 18786 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36b65112d4e40fcb9f233ee47b47949, State: Initialized, Role: FOLLOWER
I20260619 11:11:17.877074 18786 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } }
I20260619 11:11:17.877226 18786 raft_consensus.cc:388] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:11:17.877310 18786 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:11:17.877429 18786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:11:17.878239 18786 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } }
I20260619 11:11:17.878574 18786 leader_election.cc:302] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e36b65112d4e40fcb9f233ee47b47949; no voters:
I20260619 11:11:17.878856 18786 leader_election.cc:288] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:11:17.878957 18791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:11:17.879163 18791 raft_consensus.cc:686] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [term 1 LEADER]: Becoming Leader. State: Replica: e36b65112d4e40fcb9f233ee47b47949, State: Running, Role: LEADER
I20260619 11:11:17.879541 18791 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } }
I20260619 11:11:17.880061 18786 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:11:17.880950 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:17.867784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.7.90.126:46629)
0619 11:11:17.868197 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:17.868212 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:17.868246 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:11:17.868877 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:17.868894 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:17.869218 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:17.869574 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:11:17.870350 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:17.871543 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:17.872404 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:17.872904 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:17.873076 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:17.875765 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:11:17.875770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:11:17.875782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:11:17.875806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:11:17.877525 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:17.877983 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:17.877989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:17.877994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:17.878110 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:17.878394 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:17.878397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:17.878399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:17.878866 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:11:17.879650 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:17.879790 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:11:17.880039 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260619 11:11:17.881088 18792 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e36b65112d4e40fcb9f233ee47b47949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } } }
I20260619 11:11:17.881331 18792 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [sys.catalog]: This master's current role is: LEADER
I20260619 11:11:17.881227 18791 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36b65112d4e40fcb9f233ee47b47949. Latest consensus state: current_term: 1 leader_uuid: "e36b65112d4e40fcb9f233ee47b47949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36b65112d4e40fcb9f233ee47b47949" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46629 } } }
I20260619 11:11:17.881656 18791 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949 [sys.catalog]: This master's current role is: LEADER
I20260619 11:11:17.882249 18800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:11:17.882638 18786 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:11:17.887988 18800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:11:17.895442 18800 catalog_manager.cc:1347] Generated new cluster ID: 67f86e3786f344a3b71e1fd922279cca
I20260619 11:11:17.895548 18800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:11:17.907055 18800 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:11:17.908344 18800 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:11:17.913314 18800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: Generated new TSK 0
I20260619 11:11:17.914093 18800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:11:18.592288 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:56611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:11:18.711954 18831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:11:18.712234 18831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:11:18.712289 18831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:11:18.716385 18831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:11:18.716436 18831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:11:18.716517 18831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:11:18.721812 18831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.90.84:56611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:11:18.722961 18831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:11:18.723958 18831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:11:18.734921 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:18.734921 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:18.735076 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:11:18.735334 18831 server_base.cc:1034] running on GCE node
I20260619 11:11:18.735724 18831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:11:18.736328 18831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:11:18.737550 18831 hybrid_clock.cc:648] HybridClock initialized: now 1781867478737496 us; error 38 us; skew 500 ppm
Jun 19 11:11:18 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:11:18.740219 18831 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:11:18.741076 18831 webserver.cc:460] Webserver started at http://127.7.90.65:35467/ using document root <none> and password file <none>
I20260619 11:11:18.741694 18831 fs_manager.cc:359] Metadata directory not provided
I20260619 11:11:18.741780 18831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:11:18.742000 18831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:11:18 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:18.755252 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "9bcb6df3888e44b89dd6196c5486ea59"
format_stamp: "Formatted at 2026-06-19 11:11:18 on dist-test-slave-f1t5"
server_key: "0d5068007e649f2fdfd6b0886a0ca7ea"
server_key_iv: "dc1a68d2269783a730dc5115af6d3d4f"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:18.756000 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "9bcb6df3888e44b89dd6196c5486ea59"
format_stamp: "Formatted at 2026-06-19 11:11:18 on dist-test-slave-f1t5"
server_key: "0d5068007e649f2fdfd6b0886a0ca7ea"
server_key_iv: "dc1a68d2269783a730dc5115af6d3d4f"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:18.760977 18831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 11:11:18.778786 18848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:18.780496 18831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:11:18.780642 18831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "9bcb6df3888e44b89dd6196c5486ea59"
format_stamp: "Formatted at 2026-06-19 11:11:18 on dist-test-slave-f1t5"
server_key: "0d5068007e649f2fdfd6b0886a0ca7ea"
server_key_iv: "dc1a68d2269783a730dc5115af6d3d4f"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:18.780786 18831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:11:18.813210 18831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:11:18.817986 18831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:11:18.818506 18831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:11:18.819413 18831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:11:18.820741 18831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:11:18.820811 18831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:18.820868 18831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:11:18.820945 18831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 11:11:18.831746 18831 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:35753
I20260619 11:11:18.833638 18831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:11:18.833817 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 18831
I20260619 11:11:18.833941 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:11:18.834278 7529 external_mini_cluster.cc:1428] Setting key 277a422a544eb505f5fc9aa240268dc0
I20260619 11:11:18.849769 18961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:35753 every 8 connection(s)
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Jun 19 11:11:18 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:11:18.888273 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:18.852088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:46579 (local address 127.7.90.126:46629)
0619 11:11:18.859698 (+ 7610us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:18.859706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:18.859723 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:11:18.859768 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:18.859772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:18.859853 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:18.859960 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:11:18.861466 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:18.862315 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:18.864458 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:18.864834 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:18.864938 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:18.883218 (+ 18280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:11:18.883223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:11:18.883226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:11:18.883263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:11:18.885288 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:18.886142 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:18.886147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:18.886149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:18.886258 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:18.886758 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:18.886761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:18.886763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:18.887037 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:11:18.887122 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:18.887943 (+ 821us) server_negotiation.cc:299] Negotiation successful
0619 11:11:18.888087 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7517,"thread_start_us":105,"threads_started":1}
I20260619 11:11:18.890033 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:18.850868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46629 (local address 127.7.90.65:46579)
0619 11:11:18.851484 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:18.851541 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:18.852582 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:18.860288 (+ 7706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:18.860298 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:18.860639 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:11:18.861305 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:18.861322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:18.862459 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:18.862462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:18.863822 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:18.863831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:18.865465 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:18.865469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:18.865703 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:18.866990 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:11:18.867011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:11:18.882941 (+ 15930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:11:18.885480 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:18.885487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:18.885507 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:18.886011 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:18.886401 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:18.886406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:18.886411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:18.886638 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:18.887247 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:11:18.887255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:11:18.887637 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 11:11:18.888543 (+ 906us) client_negotiation.cc:240] Negotiation successful
0619 11:11:18.888914 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":176,"threads_started":1}
I20260619 11:11:18.891585 18962 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46629
I20260619 11:11:18.891855 18962 heartbeater.cc:464] Registering TS with master...
I20260619 11:11:18.892441 18962 heartbeater.cc:511] Master 127.7.90.126:46629 requested a full tablet report, sending...
I20260619 11:11:18.894260 18745 ts_manager.cc:194] Registered new tserver with Master: 9bcb6df3888e44b89dd6196c5486ea59 (127.7.90.65:35753)
I20260619 11:11:18.895776 18745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:46579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:11:18.923771 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:56611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:11:18 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 11:11:19.099156 18970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:11:19.099509 18970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:11:19.099570 18970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:11:19.105973 18970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:11:19.106046 18970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:11:19.106127 18970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:11:19.113422 18970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.90.84:56611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:11:19.114799 18970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:11:19.116117 18970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:11:19.128113 18979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:19.128193 18976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:19.128196 18977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:11:19.129279 18970 server_base.cc:1034] running on GCE node
I20260619 11:11:19.129789 18970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:11:19.130658 18970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:11:19.133957 18970 hybrid_clock.cc:648] HybridClock initialized: now 1781867479133934 us; error 36 us; skew 500 ppm
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:11:19.137671 18970 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:11:19.138789 18970 webserver.cc:460] Webserver started at http://127.7.90.66:41505/ using document root <none> and password file <none>
I20260619 11:11:19.139482 18970 fs_manager.cc:359] Metadata directory not provided
I20260619 11:11:19.139575 18970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:11:19.139814 18970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:19.156334 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "23a68a47c71c46d4bd76a8e47759abe8"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "4ed2c65884d6814d81fa5aef2f2df388"
server_key_iv: "6753e0972c95f58cc01b67ec171692fd"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.157152 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "23a68a47c71c46d4bd76a8e47759abe8"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "4ed2c65884d6814d81fa5aef2f2df388"
server_key_iv: "6753e0972c95f58cc01b67ec171692fd"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.167047 18970 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.000s
I20260619 11:11:19.197383 18986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.199627 18970 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260619 11:11:19.199769 18970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "23a68a47c71c46d4bd76a8e47759abe8"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "4ed2c65884d6814d81fa5aef2f2df388"
server_key_iv: "6753e0972c95f58cc01b67ec171692fd"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.199923 18970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:11:19.220476 18970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:11:19.224148 18970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:11:19.224377 18970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:11:19.225131 18970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:11:19.226326 18970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:11:19.226393 18970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.226522 18970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:11:19.226565 18970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.238533 18970 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:39691
I20260619 11:11:19.238603 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:39691 every 8 connection(s)
I20260619 11:11:19.239755 18970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:11:19.246344 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 18970
I20260619 11:11:19.246482 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:11:19.246752 7529 external_mini_cluster.cc:1428] Setting key 64f8ec72aefcab67abd070c50507d9a2
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:11:19.257033 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.241793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:42011 (local address 127.7.90.126:46629)
0619 11:11:19.241942 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.241946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.243073 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 11:11:19.244265 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.244268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.244329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.244426 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:11:19.246437 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.247252 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.249011 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.249375 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.249466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.252589 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:11:19.252592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:11:19.252595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:11:19.252625 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:11:19.254513 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.255159 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.255164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.255166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.255251 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.255657 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.255661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.255663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.255890 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:11:19.255967 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.256729 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.256858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:11:19.258242 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.242360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46629 (local address 127.7.90.66:42011)
0619 11:11:19.242917 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:19.242970 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:19.244021 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:19.244743 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:19.244752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:19.245580 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:11:19.246231 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.246252 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.247463 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.247467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.248851 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.248862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.249711 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.249716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.249918 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.250630 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:11:19.250653 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:11:19.252397 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:11:19.254655 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:19.254660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:19.254676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:19.255021 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:19.255364 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:19.255367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:19.255369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:19.255545 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:19.256064 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:11:19.256071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:11:19.256381 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 11:11:19.256660 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 11:11:19.256992 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260619 11:11:19.259536 19100 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46629
I20260619 11:11:19.259812 19100 heartbeater.cc:464] Registering TS with master...
I20260619 11:11:19.260411 19100 heartbeater.cc:511] Master 127.7.90.126:46629 requested a full tablet report, sending...
I20260619 11:11:19.261494 18745 ts_manager.cc:194] Registered new tserver with Master: 23a68a47c71c46d4bd76a8e47759abe8 (127.7.90.66:39691)
I20260619 11:11:19.262545 18745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:42011
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:11:19.285266 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:56611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:11:19.436280 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:11:19.436618 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:11:19.436673 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:11:19.442541 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:11:19.442616 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:11:19.442698 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:11:19.450245 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.90.84:56611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:46629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:11:19.451685 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:11:19.453076 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:11:19.468152 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:19.468629 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:11:19.469244 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:11:19.471421 19105 server_base.cc:1034] running on GCE node
I20260619 11:11:19.471985 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:11:19.472867 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:11:19.474120 19105 hybrid_clock.cc:648] HybridClock initialized: now 1781867479474052 us; error 78 us; skew 500 ppm
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:11:19.477811 19105 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:11:19.478869 19105 webserver.cc:460] Webserver started at http://127.7.90.67:44297/ using document root <none> and password file <none>
I20260619 11:11:19.479542 19105 fs_manager.cc:359] Metadata directory not provided
I20260619 11:11:19.479617 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:11:19.479830 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:19.511708 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "3c7b5b65d54a41e5ba0424b5766ab940"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "d79c05a548a1b204d2b1c86284418fbc"
server_key_iv: "fb6407a965aa9cbac411048f478fa6f8"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.512401 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "3c7b5b65d54a41e5ba0424b5766ab940"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "d79c05a548a1b204d2b1c86284418fbc"
server_key_iv: "fb6407a965aa9cbac411048f478fa6f8"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.517596 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 11:11:19.534494 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.536312 19105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:11:19.536455 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "3c7b5b65d54a41e5ba0424b5766ab940"
format_stamp: "Formatted at 2026-06-19 11:11:19 on dist-test-slave-f1t5"
server_key: "d79c05a548a1b204d2b1c86284418fbc"
server_key_iv: "fb6407a965aa9cbac411048f478fa6f8"
server_key_version: "kuduclusterkey@0"
I20260619 11:11:19.536609 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:11:19.563334 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:11:19.566954 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:11:19.567158 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:11:19.567893 19105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:11:19.568964 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:11:19.569027 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.569108 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:11:19.569141 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:11:19.581161 19105 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:39267
I20260619 11:11:19.581246 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:39267 every 8 connection(s)
I20260619 11:11:19.582407 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:11:19.592509 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 19105
I20260619 11:11:19.592662 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:11:19.593063 7529 external_mini_cluster.cc:1428] Setting key fdb62f8f628b982ef89be248ae6ba596
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:11:19.598757 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.584536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:60387 (local address 127.7.90.126:46629)
0619 11:11:19.584679 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.584684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.585820 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 11:11:19.587183 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.587187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.587247 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.587307 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:11:19.588992 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.589932 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.591668 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.591907 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.592031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.594993 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:11:19.594997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:11:19.594999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:11:19.595036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:11:19.596710 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.597203 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.597207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.597209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.597297 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.597668 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.597671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.597673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.597868 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:11:19.597966 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.598462 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.598588 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:11:19.599575 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.584999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:46629 (local address 127.7.90.67:60387)
0619 11:11:19.585656 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:19.585712 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:19.586922 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:19.587756 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:19.587766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:19.588134 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:11:19.588805 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.588827 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.590077 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.590081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.591504 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.591522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.592066 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.592071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.592310 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.593027 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:11:19.593051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:11:19.594830 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:11:19.596823 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:19.596827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:19.596839 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:19.597098 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:19.597385 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:11:19.597387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:11:19.597389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:11:19.597577 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:11:19.597965 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:11:19.597970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:11:19.598209 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 11:11:19.598357 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:11:19.598596 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":136,"threads_started":1}
I20260619 11:11:19.600694 19234 heartbeater.cc:347] Connected to a master server at 127.7.90.126:46629
I20260619 11:11:19.600930 19234 heartbeater.cc:464] Registering TS with master...
I20260619 11:11:19.601451 19234 heartbeater.cc:511] Master 127.7.90.126:46629 requested a full tablet report, sending...
I20260619 11:11:19.602496 18745 ts_manager.cc:194] Registered new tserver with Master: 3c7b5b65d54a41e5ba0424b5766ab940 (127.7.90.67:39267)
I20260619 11:11:19.603075 18745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:60387
I20260619 11:11:19.607213 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:11:19.617475 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.609348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.7.90.126:46629)
0619 11:11:19.609494 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.609499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.609762 (+ 263us) server_negotiation.cc:407] Connection header received
0619 11:11:19.609841 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.609844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.609892 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.609974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:11:19.610629 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.611410 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.612330 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.612641 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.612746 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.614210 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:11:19.614213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:11:19.614216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:11:19.614243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:11:19.615769 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.616158 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.616162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.616164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.616252 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:11:19.616529 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:11:19.616533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:11:19.616536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:11:19.616775 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:11:19.616913 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.617117 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.617263 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:11:19.621497 18745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:11:19.625919 18745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:11:19.643579 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.635314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35753 (local address 127.0.0.1:36604)
0619 11:11:19.637666 (+ 2352us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:19.637680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:19.637774 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:19.638521 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:19.638524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:19.638537 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:11:19.638625 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.638633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.639915 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.639919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.641264 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.641272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.643009 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.643015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.643156 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.643199 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 11:11:19.643306 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:11:19.643430 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2294,"thread_start_us":91,"threads_started":1}
I20260619 11:11:19.644423 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.635691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.7.90.65:35753)
0619 11:11:19.637987 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.637994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.638012 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:11:19.638091 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.638096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.638270 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.638394 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:11:19.638744 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.639799 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.641396 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.642703 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.642801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.642946 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.644176 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.644285 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"thread_start_us":67,"threads_started":1}
I20260619 11:11:19.644874 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:39267 (local address 127.0.0.1:41254)
0619 11:11:19.636434 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:19.636481 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:19.636610 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:19.638385 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:19.638390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:19.638411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:11:19.638534 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.638540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.640835 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.640838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.642010 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.642019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.644484 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.644490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.644595 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.644615 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:11:19.644684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:11:19.644751 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":75,"threads_started":1}
I20260619 11:11:19.645481 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.635691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.7.90.67:39267)
0619 11:11:19.637455 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.637741 (+ 286us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.637766 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:11:19.637849 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.637855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.638068 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.638213 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:11:19.638673 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.640715 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.642652 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.644179 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.644275 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.644422 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.645239 (+ 817us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.645333 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":150,"threads_started":1}
I20260619 11:11:19.647786 18888 tablet_service.cc:1467] Processing CreateTablet for tablet be56a2dd204f463ea61af4b6c18a341d (DEFAULT_TABLE table=test-table [id=f8812c87065a4d68ad81955b097a4f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:11:19.647976 19168 tablet_service.cc:1467] Processing CreateTablet for tablet be56a2dd204f463ea61af4b6c18a341d (DEFAULT_TABLE table=test-table [id=f8812c87065a4d68ad81955b097a4f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:11:19.648844 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be56a2dd204f463ea61af4b6c18a341d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:11:19.648869 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be56a2dd204f463ea61af4b6c18a341d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:11:19.656553 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.636111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:39691 (local address 127.0.0.1:53108)
0619 11:11:19.639824 (+ 3713us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:19.639838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:19.639952 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:19.640835 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:19.640838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:19.640857 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:11:19.640958 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.640965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.653607 (+ 12642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.653615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.654813 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:19.654825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.656137 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.656141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:19.656236 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.656259 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:11:19.656338 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:11:19.656412 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3653,"thread_start_us":87,"threads_started":1}
I20260619 11:11:19.656881 19253 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Bootstrap starting.
I20260619 11:11:19.657032 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:19.636486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.7.90.66:39691)
0619 11:11:19.637698 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:19.637714 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:19.640052 (+ 2338us) server_negotiation.cc:407] Connection header received
0619 11:11:19.640134 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:19.640139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:19.640328 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:19.640476 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:11:19.641092 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.642243 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.655021 (+ 12778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:19.655821 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:19.655937 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:19.656082 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:19.656779 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 11:11:19.656876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":123,"threads_started":1}
I20260619 11:11:19.659600 19253 tablet_bootstrap.cc:654] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Neither blocks nor log segments found. Creating new log.
I20260619 11:11:19.660112 19252 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Bootstrap starting.
I20260619 11:11:19.659844 19034 tablet_service.cc:1467] Processing CreateTablet for tablet be56a2dd204f463ea61af4b6c18a341d (DEFAULT_TABLE table=test-table [id=f8812c87065a4d68ad81955b097a4f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:11:19.660416 19253 log.cc:824] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Log is configured to *not* fsync() on all Append() calls
I20260619 11:11:19.660876 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be56a2dd204f463ea61af4b6c18a341d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:11:19.663492 19253 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: No bootstrap required, opened a new log
I20260619 11:11:19.663703 19253 ts_tablet_manager.cc:1397] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 11:11:19.663892 19252 tablet_bootstrap.cc:654] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Neither blocks nor log segments found. Creating new log.
I20260619 11:11:19.664812 19252 log.cc:824] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Log is configured to *not* fsync() on all Append() calls
I20260619 11:11:19.667529 19253 raft_consensus.cc:348] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.667793 19253 raft_consensus.cc:374] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:11:19.668942 19253 raft_consensus.cc:729] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7b5b65d54a41e5ba0424b5766ab940, State: Initialized, Role: FOLLOWER
I20260619 11:11:19.669425 19253 consensus_queue.cc:260] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.670042 19252 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: No bootstrap required, opened a new log
I20260619 11:11:19.670238 19252 ts_tablet_manager.cc:1397] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 11:11:19.670428 19253 ts_tablet_manager.cc:1428] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:11:19.670804 19234 heartbeater.cc:503] Master 127.7.90.126:46629 was elected leader, sending a full tablet report...
I20260619 11:11:19.673956 19255 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Bootstrap starting.
I20260619 11:11:19.674898 19252 raft_consensus.cc:348] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.675153 19252 raft_consensus.cc:374] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:11:19.675202 19252 raft_consensus.cc:729] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcb6df3888e44b89dd6196c5486ea59, State: Initialized, Role: FOLLOWER
I20260619 11:11:19.675679 19252 consensus_queue.cc:260] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.676766 19252 ts_tablet_manager.cc:1428] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:11:19.677115 18962 heartbeater.cc:503] Master 127.7.90.126:46629 was elected leader, sending a full tablet report...
I20260619 11:11:19.677284 19255 tablet_bootstrap.cc:654] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Neither blocks nor log segments found. Creating new log.
I20260619 11:11:19.678093 19255 log.cc:824] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:11:19.680881 19255 tablet_bootstrap.cc:492] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: No bootstrap required, opened a new log
I20260619 11:11:19.681170 19255 ts_tablet_manager.cc:1397] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:11:19.684144 19255 raft_consensus.cc:348] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.684369 19255 raft_consensus.cc:374] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:11:19.684455 19255 raft_consensus.cc:729] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a68a47c71c46d4bd76a8e47759abe8, State: Initialized, Role: FOLLOWER
I20260619 11:11:19.684924 19255 consensus_queue.cc:260] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:19.685494 19100 heartbeater.cc:503] Master 127.7.90.126:46629 was elected leader, sending a full tablet report...
I20260619 11:11:19.685905 19255 ts_tablet_manager.cc:1428] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:11:19.742813 19101 tablet.cc:2369] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 11:11:19 dist-test-slave-f1t5 krb5kdc[18274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
W20260619 11:11:19.834506 19235 tablet.cc:2369] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:11:19.853713 18963 tablet.cc:2369] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:11:20.059010 19260 raft_consensus.cc:482] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:11:20.059270 19260 raft_consensus.cc:504] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:20.060619 19260 leader_election.cc:288] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c7b5b65d54a41e5ba0424b5766ab940 (127.7.90.67:39267), 9bcb6df3888e44b89dd6196c5486ea59 (127.7.90.65:35753)
I20260619 11:11:20.066215 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.061376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:52769 (local address 127.7.90.65:35753)
0619 11:11:20.061505 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:20.061509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:20.061629 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:11:20.062041 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:20.062045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:20.062100 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:20.062182 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:11:20.062500 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.063251 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.064876 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.065465 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.065602 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:20.065668 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:20.066005 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:11:20.066081 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:11:20.066668 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.061010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35753 (local address 127.7.90.66:52769)
0619 11:11:20.061494 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:20.061505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:20.061902 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:20.062217 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:20.062220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:20.062242 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:11:20.062371 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:20.062378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.063386 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.063390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:20.064726 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:20.064737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.065763 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.065767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:20.065863 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:20.065888 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:11:20.066427 (+ 539us) client_negotiation.cc:240] Negotiation successful
0619 11:11:20.066506 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":30,"thread_start_us":73,"threads_started":1}
I20260619 11:11:20.066972 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.061230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:48223 (local address 127.7.90.67:39267)
0619 11:11:20.061364 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:20.061370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:20.061386 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:11:20.061447 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:20.061450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:20.061499 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:20.061602 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:11:20.062846 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.063693 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.065433 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.066166 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.066258 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:20.066333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:20.066728 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 11:11:20.066824 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:11:20.067380 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.060807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:39267 (local address 127.7.90.66:48223)
0619 11:11:20.061152 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:20.061167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:11:20.061279 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:11:20.062535 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:11:20.062538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:11:20.062561 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:11:20.062697 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:20.062707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.063859 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.063862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:20.065278 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:11:20.065291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.066426 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.066429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:11:20.066553 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:20.066582 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:11:20.067169 (+ 587us) client_negotiation.cc:240] Negotiation successful
0619 11:11:20.067239 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":81,"threads_started":1}
I20260619 11:11:20.067353 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be56a2dd204f463ea61af4b6c18a341d" candidate_uuid: "23a68a47c71c46d4bd76a8e47759abe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcb6df3888e44b89dd6196c5486ea59" is_pre_election: true
I20260619 11:11:20.067721 18907 raft_consensus.cc:2454] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a68a47c71c46d4bd76a8e47759abe8 in term 0.
I20260619 11:11:20.068089 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be56a2dd204f463ea61af4b6c18a341d" candidate_uuid: "23a68a47c71c46d4bd76a8e47759abe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" is_pre_election: true
I20260619 11:11:20.068421 19188 raft_consensus.cc:2454] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23a68a47c71c46d4bd76a8e47759abe8 in term 0.
I20260619 11:11:20.069060 18988 leader_election.cc:302] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a68a47c71c46d4bd76a8e47759abe8, 3c7b5b65d54a41e5ba0424b5766ab940; no voters:
I20260619 11:11:20.069660 19260 raft_consensus.cc:2781] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:11:20.069759 19260 raft_consensus.cc:482] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:11:20.069811 19260 raft_consensus.cc:3037] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:11:20.070863 19260 raft_consensus.cc:504] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:20.071259 19260 leader_election.cc:288] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [CANDIDATE]: Term 1 election: Requested vote from peers 3c7b5b65d54a41e5ba0424b5766ab940 (127.7.90.67:39267), 9bcb6df3888e44b89dd6196c5486ea59 (127.7.90.65:35753)
I20260619 11:11:20.071573 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be56a2dd204f463ea61af4b6c18a341d" candidate_uuid: "23a68a47c71c46d4bd76a8e47759abe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcb6df3888e44b89dd6196c5486ea59"
I20260619 11:11:20.071712 18907 raft_consensus.cc:3037] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:11:20.072642 18907 raft_consensus.cc:2454] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a68a47c71c46d4bd76a8e47759abe8 in term 1.
I20260619 11:11:20.072944 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be56a2dd204f463ea61af4b6c18a341d" candidate_uuid: "23a68a47c71c46d4bd76a8e47759abe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7b5b65d54a41e5ba0424b5766ab940"
I20260619 11:11:20.072971 18990 leader_election.cc:302] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a68a47c71c46d4bd76a8e47759abe8, 9bcb6df3888e44b89dd6196c5486ea59; no voters:
I20260619 11:11:20.073105 19188 raft_consensus.cc:3037] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:11:20.073180 19260 raft_consensus.cc:2781] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:11:20.073383 19260 raft_consensus.cc:686] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 1 LEADER]: Becoming Leader. State: Replica: 23a68a47c71c46d4bd76a8e47759abe8, State: Running, Role: LEADER
I20260619 11:11:20.073814 19260 consensus_queue.cc:237] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } }
I20260619 11:11:20.074316 19188 raft_consensus.cc:2454] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23a68a47c71c46d4bd76a8e47759abe8 in term 1.
I20260619 11:11:20.077365 18745 catalog_manager.cc:5495] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23a68a47c71c46d4bd76a8e47759abe8 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "23a68a47c71c46d4bd76a8e47759abe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23a68a47c71c46d4bd76a8e47759abe8" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 39691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 } health_report { overall_health: UNKNOWN } } }
I20260619 11:11:20.094663 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.090999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.7.90.66:39691)
0619 11:11:20.091139 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:11:20.091144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:11:20.091159 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:11:20.091281 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:11:20.091284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:11:20.091336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:11:20.091449 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:11:20.091695 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.092214 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.093086 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:11:20.093306 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:11:20.093420 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:11:20.093846 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:11:20.094266 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:11:20.094393 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:11:20.094459 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 11:11:20.094515 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:11:20.100919 19188 raft_consensus.cc:1261] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 1 FOLLOWER]: Refusing update from remote peer 23a68a47c71c46d4bd76a8e47759abe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:11:20.100919 18907 raft_consensus.cc:1261] T be56a2dd204f463ea61af4b6c18a341d P 9bcb6df3888e44b89dd6196c5486ea59 [term 1 FOLLOWER]: Refusing update from remote peer 23a68a47c71c46d4bd76a8e47759abe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:11:20.101699 19260 consensus_queue.cc:1035] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7b5b65d54a41e5ba0424b5766ab940" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 39267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:11:20.101907 19272 consensus_queue.cc:1035] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcb6df3888e44b89dd6196c5486ea59" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:11:20.108558 19275 mvcc.cc:205] Tried to move back new op lower bound from 7298529198485848064 to 7298529198385369088. Current Snapshot: MvccSnapshot[applied={T|T < 7298529198485848064}]
I20260619 11:11:20.113463 19277 mvcc.cc:205] Tried to move back new op lower bound from 7298529198485848064 to 7298529198385369088. Current Snapshot: MvccSnapshot[applied={T|T < 7298529198485848064}]
I20260619 11:11:20.114634 19276 mvcc.cc:205] Tried to move back new op lower bound from 7298529198485848064 to 7298529198385369088. Current Snapshot: MvccSnapshot[applied={T|T < 7298529198485848064 or (T in {7298529198485848064})}]
I20260619 11:11:20.122395 18750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:11:20.122570 18750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:11:20.125078 18750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e36b65112d4e40fcb9f233ee47b47949: Sending DeleteTablet for 3 replicas of tablet be56a2dd204f463ea61af4b6c18a341d
I20260619 11:11:20.125841 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet be56a2dd204f463ea61af4b6c18a341d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:11:20 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:41254
I20260619 11:11:20.125947 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet be56a2dd204f463ea61af4b6c18a341d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:11:20 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:36604
I20260619 11:11:20.126677 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet be56a2dd204f463ea61af4b6c18a341d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:11:20 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:53108
I20260619 11:11:20.127163 19286 tablet_replica.cc:330] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: stopping tablet replica
I20260619 11:11:20.127483 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 18831
I20260619 11:11:20.127508 19286 raft_consensus.cc:2229] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:11:20.129985 19286 raft_consensus.cc:2258] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:11:20.131834 19286 ts_tablet_manager.cc:1905] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:11:20.132319 19284 tablet_replica.cc:330] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: stopping tablet replica
I20260619 11:11:20.132659 19284 raft_consensus.cc:2229] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:11:20.132972 19284 raft_consensus.cc:2258] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:11:20.134859 19286 ts_tablet_manager.cc:1918] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:11:20.135001 19286 log.cc:1196] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/be56a2dd204f463ea61af4b6c18a341d
I20260619 11:11:20.135396 19286 ts_tablet_manager.cc:1939] T be56a2dd204f463ea61af4b6c18a341d P 23a68a47c71c46d4bd76a8e47759abe8: Deleting consensus metadata
I20260619 11:11:20.136636 18736 catalog_manager.cc:4841] TS 23a68a47c71c46d4bd76a8e47759abe8 (127.7.90.66:39691): tablet be56a2dd204f463ea61af4b6c18a341d (table test-table [id=f8812c87065a4d68ad81955b097a4f92]) successfully deleted
I20260619 11:11:20.137697 19284 ts_tablet_manager.cc:1905] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:11:20.140172 19284 ts_tablet_manager.cc:1918] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:11:20.140264 19284 log.cc:1196] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/be56a2dd204f463ea61af4b6c18a341d
I20260619 11:11:20.140573 19284 ts_tablet_manager.cc:1939] T be56a2dd204f463ea61af4b6c18a341d P 3c7b5b65d54a41e5ba0424b5766ab940: Deleting consensus metadata
I20260619 11:11:20.141710 18735 catalog_manager.cc:4841] TS 3c7b5b65d54a41e5ba0424b5766ab940 (127.7.90.67:39267): tablet be56a2dd204f463ea61af4b6c18a341d (table test-table [id=f8812c87065a4d68ad81955b097a4f92]) successfully deleted
W20260619 11:11:20.141752 18737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:35753 (error 108)
I20260619 11:11:20.142609 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 18970
I20260619 11:11:20.143285 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:11:20.142481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35753 (local address 127.0.0.1:36614)
0619 11:11:20.143133 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 11:11:20.143202 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:35753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561}
W20260619 11:11:20.143492 18737 catalog_manager.cc:4584] TS 9bcb6df3888e44b89dd6196c5486ea59 (127.7.90.65:35753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be56a2dd204f463ea61af4b6c18a341d: Network error: Client connection negotiation failed: client connection to 127.7.90.65:35753: connect: Connection refused (error 111)
I20260619 11:11:20.148695 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 19105
I20260619 11:11:20.154340 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 18718
I20260619 11:11:20.161931 7529 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:11:20.497762 7529 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:11:20.497927 7529 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:11:20.843986 7529 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:11:20.844 UTC [18304] LOG: received smart shutdown request
2026-06-19 11:11:20.847 UTC [18304] LOG: background worker "logical replication launcher" (PID 18313) exited with exit code 1
2026-06-19 11:11:20.847 UTC [18308] LOG: shutting down
2026-06-19 11:11:20.981 UTC [18304] LOG: database system is shut down
2026-06-19T11:11:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43941 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:11:21 dist-test-slave-f1t5 krb5kdc[19296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:11:21 dist-test-slave-f1t5 krb5kdc[19296](info): set up 2 sockets
Jun 19 11:11:21 dist-test-slave-f1t5 krb5kdc[19296](info): commencing operation
krb5kdc: starting...
W20260619 11:11:22.129633 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:11:22 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:11:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangeradmin_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangeradmin_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangerlookup_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangerlookup_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
I20260619 11:11:22.291956 7529 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7DZYiV/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/postgres -l logfile start
2026-06-19 11:11:24.546 UTC [19326] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:11:24.546 UTC [19326] LOG: listening on IPv4 address "127.7.90.84", port 40815
2026-06-19 11:11:24.549 UTC [19326] LOG: listening on Unix socket "/tmp/.s.PGSQL.40815"
2026-06-19 11:11:24.583 UTC [19329] LOG: database system was shut down at 2026-06-19 11:11:23 UTC
2026-06-19 11:11:24.587 UTC [19326] LOG: database system is ready to accept connections
I20260619 11:11:25.525447 7529 mini_postgres.cc:96] Postgres bound to 40815
2026-06-19 11:11:25.531 UTC [19337] FATAL: database "slave" does not exist
127.7.90.84:40815 - accepting connections
I20260619 11:11:25.532083 7529 mini_ranger.cc:161] Starting Ranger...
I20260619 11:11:25.546317 7529 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:11:25.953454 7529 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:11:25.953590 7529 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin
2026-06-19 11:11:26,220 [I] DB FLAVOR :POSTGRES
2026-06-19 11:11:26,220 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:11:26,220 [I] Checking connection..
2026-06-19 11:11:26,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:11:26,475 [I] Checking connection passed.
2026-06-19 11:11:26,475 [I] --------- Verifying version history table ---------
2026-06-19 11:11:26,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:11:26,732 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:11:26,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:11:26,980 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:11:26,980 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:11:26,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:11:27.232 UTC [19401] WARNING: there is no transaction in progress
2026-06-19 11:11:27,237 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:11:27,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:11:27,492 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:11:27,492 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:11:27,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:11:27,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:11:27,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f1t5.c.gcp-upstream.internal','N') ;"
2026-06-19 11:11:28,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:11:28,425 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:11:28,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:11:28,654 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:11:28,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:11:28,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:11:29,145 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:11:29,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:11:30.022 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.037 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.052 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.066 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.081 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.098 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.149 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.155 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.168 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.178 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.187 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.200 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.209 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.215 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.221 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30.227 UTC [19509] WARNING: there is no transaction in progress
2026-06-19 11:11:30,786 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:11:30,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:11:31,023 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:11:31,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:11:31,273 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:11:31,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:11:31,492 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:11:31,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:11:31,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:11:31,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f1t5.c.gcp-upstream.internal';"
2026-06-19 11:11:32,191 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:11:32,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:11:32,409 [I] DB_PATCHES have already been applied
I20260619 11:11:32.415107 7529 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7DZYiV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:11:32.415162 7529 mini_ranger.cc:193] Using host: 127.7.90.84
I20260619 11:11:32.418604 7529 mini_ranger.cc:227] Ranger admin URL: http://127.7.90.84:37161
Jun 19, 2026 11:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:11:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:11:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:11:33.523485 7529 mini_ranger.cc:160] Time spent starting Ranger: real 7.991s user 0.002s sys 0.007s
Jun 19, 2026 11:11:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:11:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 11:11:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:11:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:11:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:11:53.825103 7529 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.90.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.90.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangerkms_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.90.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/rangerkms_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.90.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.90.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.90.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/HTTP_127.7.90.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:11:53 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867513, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:11:53.918195 7529 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:11:53.928073 7529 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:11:54.697245 7529 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:11:54.697326 7529 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms
I20260619 11:11:54.697350 7529 mini_ranger_kms.cc:227] Using postgres at 127.7.90.84:40815
2026-06-19 11:11:54,876 [I] DB FLAVOR :POSTGRES
2026-06-19 11:11:54,876 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:11:54,877 [I] Checking connection
2026-06-19 11:11:54,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:11:55,140 [I] connection success
2026-06-19 11:11:55,140 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:11:55,140 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:11:55,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:11:55,397 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:11:55,397 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:11:55,397 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:11:55,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.90.84:40815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:11:55,682 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:11:56.174921 7529 mini_ranger_kms.cc:310] Created kms service
I20260619 11:11:56.380905 7529 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:11:56.423422 7529 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:11:56 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867513, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:11:56.800714 7529 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:11:56.800863 7529 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms:/tmp/dist-test-task7DZYiV/build/debug/bin/postgresql.jar:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7DZYiV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:11:56.800884 7529 mini_ranger_kms.cc:240] Using host: 127.7.90.84
I20260619 11:11:56.805576 7529 mini_ranger_kms.cc:276] Ranger KMS PID: 19689
I20260619 11:11:56.805665 7529 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.90.84:38519
Jun 19, 2026 11:11:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:11:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:11:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:11:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:11:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:11:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 11:11:58 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867518, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:11:59.057873 7529 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.140s user 0.025s sys 0.127s
I20260619 11:11:59.058009 7529 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:11:59.058090 7529 mini_ranger_kms.cc:401] 127.7.90.84:38519/kms/v1/keys
Jun 19 11:11:59 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867518, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
Jun 19 11:11:59 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867518, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.90.84@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
Jun 19 11:12:01 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:12:01.864380 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37811
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37811
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:01.985348 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:01.985656 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:01.985728 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:01.991058 19740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:01.991119 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:01.991139 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:01.991180 19740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:01.991209 19740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:01.996783 19740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.90.84:38519/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37811
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:01.997910 19740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:01.998930 19740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:02.007488 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:02.007489 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:02.007614 19740 server_base.cc:1034] running on GCE node
W20260619 11:12:02.007472 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:02.008239 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:02.009214 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:02.010363 19740 hybrid_clock.cc:648] HybridClock initialized: now 1781867522010339 us; error 37 us; skew 500 ppm
Jun 19 11:12:02 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:02.017694 19740 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:02.018677 19740 webserver.cc:460] Webserver started at http://127.7.90.126:34723/ using document root <none> and password file <none>
I20260619 11:12:02.019258 19740 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:02.019342 19740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:02.019574 19740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:12:02 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:02.067116 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "ea75f218761f42c497ebd265ae1f14b9"
format_stamp: "Formatted at 2026-06-19 11:12:02 on dist-test-slave-f1t5"
server_key: "42716b17d5ea7cbfac157505aeec6d74"
server_key_iv: "02886e13d7b1f3612ffc290b1daae7c9"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:02.067744 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "ea75f218761f42c497ebd265ae1f14b9"
format_stamp: "Formatted at 2026-06-19 11:12:02 on dist-test-slave-f1t5"
server_key: "42716b17d5ea7cbfac157505aeec6d74"
server_key_iv: "02886e13d7b1f3612ffc290b1daae7c9"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:02.071791 19740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:12:02.085635 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:02.087072 19740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:12:02.087185 19740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "ea75f218761f42c497ebd265ae1f14b9"
format_stamp: "Formatted at 2026-06-19 11:12:02 on dist-test-slave-f1t5"
server_key: "42716b17d5ea7cbfac157505aeec6d74"
server_key_iv: "02886e13d7b1f3612ffc290b1daae7c9"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:02.087343 19740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:02.116837 19740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:02.119671 19740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:02.119868 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:02.127921 19740 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37811
I20260619 11:12:02.127947 19807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37811 every 8 connection(s)
I20260619 11:12:02.128857 19740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:02.131608 19808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:02.132820 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 19740
I20260619 11:12:02.132949 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:12:02.133275 7529 external_mini_cluster.cc:1428] Setting key 685b413dffc05695863f5f2f84c6475e
I20260619 11:12:02.138501 19808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Bootstrap starting.
I20260619 11:12:02.140954 19808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:02.141824 19808 log.cc:824] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:02.143903 19808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: No bootstrap required, opened a new log
Jun 19 11:12:02 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:02.148564 19808 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:02.148854 19808 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:02.148952 19808 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea75f218761f42c497ebd265ae1f14b9, State: Initialized, Role: FOLLOWER
I20260619 11:12:02.149430 19808 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:02.149592 19808 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:02.149674 19808 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:02.149773 19808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:02.150799 19808 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:02.151146 19808 leader_election.cc:302] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea75f218761f42c497ebd265ae1f14b9; no voters:
I20260619 11:12:02.151477 19808 leader_election.cc:288] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:12:02.151561 19813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:02.151769 19813 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 LEADER]: Becoming Leader. State: Replica: ea75f218761f42c497ebd265ae1f14b9, State: Running, Role: LEADER
I20260619 11:12:02.151790 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:02.134783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.7.90.126:37811)
0619 11:12:02.135239 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:02.135258 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:02.135299 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:12:02.136185 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:02.136212 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:02.136660 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:02.137040 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:02.137987 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:02.139395 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:02.141487 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:02.142232 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:02.142387 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:02.145412 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:02.145422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:02.145455 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:02.145493 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:02.147873 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:02.148408 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:02.148414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:02.148420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:02.148559 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:02.148922 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:02.148926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:02.148928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:02.149458 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:02.150300 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:02.150457 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 11:12:02.150756 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260619 11:12:02.152269 19813 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:02.152783 19808 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:02.153563 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea75f218761f42c497ebd265ae1f14b9. Latest consensus state: current_term: 1 leader_uuid: "ea75f218761f42c497ebd265ae1f14b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } } }
I20260619 11:12:02.153743 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:02.153990 19814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea75f218761f42c497ebd265ae1f14b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } } }
I20260619 11:12:02.154137 19814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:02.154493 19817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:02.157763 19817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:02.158425 19808 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:12:02.166630 19817 catalog_manager.cc:1347] Generated new cluster ID: 6cc4c83d8a8443419f8f4183261e6b56
I20260619 11:12:02.166716 19817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:02.184656 19817 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:12:02.186059 19817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:02.193879 19817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Generated new TSK 0
I20260619 11:12:02.194664 19817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:12:02.884500 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:03.024129 19853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:03.024374 19853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:03.024412 19853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:03.030144 19853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:03.030215 19853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:03.030298 19853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:03.037750 19853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:03.038964 19853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:03.040269 19853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:03.049829 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.049932 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.049832 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:03.051079 19853 server_base.cc:1034] running on GCE node
I20260619 11:12:03.051478 19853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:03.052006 19853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:03.055284 19853 hybrid_clock.cc:648] HybridClock initialized: now 1781867523055227 us; error 54 us; skew 500 ppm
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:03.064087 19853 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:03.065490 19853 webserver.cc:460] Webserver started at http://127.7.90.65:35935/ using document root <none> and password file <none>
I20260619 11:12:03.066265 19853 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:03.066406 19853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:03.066680 19853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:03.095983 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "17c28b4b865248b1ac9007249683f1f9"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "25fbbdef5d1e827764a339a101b3ae96"
server_key_iv: "0e5e7ec63e5d42610384a421b8e60436"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.096897 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "17c28b4b865248b1ac9007249683f1f9"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "25fbbdef5d1e827764a339a101b3ae96"
server_key_iv: "0e5e7ec63e5d42610384a421b8e60436"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.101912 19853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 11:12:03.149958 19872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.151808 19853 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.002s sys 0.000s
I20260619 11:12:03.151953 19853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "17c28b4b865248b1ac9007249683f1f9"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "25fbbdef5d1e827764a339a101b3ae96"
server_key_iv: "0e5e7ec63e5d42610384a421b8e60436"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.152099 19853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:03.183939 19853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:03.188506 19853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:03.188832 19853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:03.189781 19853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:03.191094 19853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:03.191229 19853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 11:12:03.191401 19853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:03.191509 19853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.202931 19853 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:35299
I20260619 11:12:03.204187 19853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:03.206223 19985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:35299 every 8 connection(s)
I20260619 11:12:03.207247 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 19853
I20260619 11:12:03.207370 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:12:03.207703 7529 external_mini_cluster.cc:1428] Setting key 0fd197c57734a85d4e89138b2b9984bc
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:03.224347 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.209817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:33593 (local address 127.7.90.126:37811)
0619 11:12:03.212071 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:03.212076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:03.212090 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:12:03.212129 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:03.212132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:03.212176 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:03.212271 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:03.213818 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.214669 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.216371 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.216703 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.216776 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.219713 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:03.219717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:03.219720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:03.219745 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:03.221659 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.222304 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.222317 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.222322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.222414 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.222846 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.222851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.222854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.223116 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:03.223202 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:03.223907 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 11:12:03.224050 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":88,"threads_started":1}
I20260619 11:12:03.226116 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.208648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.65:33593)
0619 11:12:03.210578 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:03.210631 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:03.211664 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:03.212580 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:03.212589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:03.212941 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:03.213654 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.213673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.214813 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.214817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.216215 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.216226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.216967 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.216971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.217154 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.217878 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:03.217898 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:03.219533 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:03.221796 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.221805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.221821 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.222170 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.222524 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.222529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.222532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.222729 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.223277 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:03.223286 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:03.223628 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 11:12:03.224600 (+ 972us) client_negotiation.cc:240] Negotiation successful
0619 11:12:03.224956 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"thread_start_us":138,"threads_started":1}
I20260619 11:12:03.228338 19986 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:03.228605 19986 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:12:03.229216 19986 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:03.231002 19772 ts_manager.cc:194] Registered new tserver with Master: 17c28b4b865248b1ac9007249683f1f9 (127.7.90.65:35299)
I20260619 11:12:03.232430 19772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:33593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:12:03.249123 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:03.402559 19992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:03.402967 19992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:03.403085 19992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:03.408952 19992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:03.409027 19992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:03.409109 19992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:03.414453 19992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:03.415484 19992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:03.416558 19992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:03.427235 20001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.427341 19999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.427243 19998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:03.427621 19992 server_base.cc:1034] running on GCE node
I20260619 11:12:03.428104 19992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:03.429311 19992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:03.430751 19992 hybrid_clock.cc:648] HybridClock initialized: now 1781867523430719 us; error 43 us; skew 500 ppm
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:03.434641 19992 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:03.435497 19992 webserver.cc:460] Webserver started at http://127.7.90.66:35463/ using document root <none> and password file <none>
I20260619 11:12:03.436031 19992 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:03.436129 19992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:03.436378 19992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:03.450986 19992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "65ce7309b5ec46cf93699d439ad7d09c"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "d9eb42e852eed35325d765fab4b9bc59"
server_key_iv: "4cff36f94b9870a1556beeab8396de9a"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.451742 19992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "65ce7309b5ec46cf93699d439ad7d09c"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "d9eb42e852eed35325d765fab4b9bc59"
server_key_iv: "4cff36f94b9870a1556beeab8396de9a"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.474831 19992 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.004s sys 0.000s
I20260619 11:12:03.501906 20008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.504598 19992 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260619 11:12:03.504810 19992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "65ce7309b5ec46cf93699d439ad7d09c"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "d9eb42e852eed35325d765fab4b9bc59"
server_key_iv: "4cff36f94b9870a1556beeab8396de9a"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.505162 19992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:03.530416 19992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:03.534049 19992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:03.534291 19992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:03.534974 19992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:03.536032 19992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:03.536137 19992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.536233 19992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:03.536294 19992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.549404 19992 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35231
I20260619 11:12:03.550592 19992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:03.551980 20121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35231 every 8 connection(s)
I20260619 11:12:03.552953 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 19992
I20260619 11:12:03.553084 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:12:03.553431 7529 external_mini_cluster.cc:1428] Setting key f3c168c278c4f9790ffd4fd09e939673
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:03.570349 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.556156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:47623 (local address 127.7.90.126:37811)
0619 11:12:03.556277 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:03.556282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:03.556742 (+ 460us) server_negotiation.cc:407] Connection header received
0619 11:12:03.557895 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:03.557898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:03.557952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:03.558028 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:03.559371 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.560193 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.561921 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.562280 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.562365 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.566087 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:03.566091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:03.566094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:03.566125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:03.567864 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.568514 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.568517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.568519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.568601 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.568981 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.568984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.568986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.569201 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:03.569277 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:03.570031 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 11:12:03.570162 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:12:03.572170 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.554696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.66:47623)
0619 11:12:03.556608 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:03.556663 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:03.557678 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:03.558224 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:03.558233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:03.558578 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:03.559198 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.559214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.560356 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.560360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.561787 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.561797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.562554 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.562557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.562744 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.563422 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:03.563441 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:03.565881 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:03.568017 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.568024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.568042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.568398 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.568701 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.568703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.568705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.568878 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.569354 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:03.569360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:03.569742 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 11:12:03.570808 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0619 11:12:03.571154 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"thread_start_us":135,"threads_started":1}
I20260619 11:12:03.573684 20122 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:03.573920 20122 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:03.574460 20122 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:03.575538 19772 ts_manager.cc:194] Registered new tserver with Master: 65ce7309b5ec46cf93699d439ad7d09c (127.7.90.66:35231)
I20260619 11:12:03.576236 19772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:47623
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:12:03.612726 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
W20260619 11:12:03.804777 20127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:03.805146 20127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:03.805208 20127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:03.811151 20127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:03.811229 20127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:03.811316 20127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:03.819031 20127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:03.820389 20127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:03.821666 20127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:03.833482 20133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.833690 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:03.833631 20135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:03.837180 20127 server_base.cc:1034] running on GCE node
I20260619 11:12:03.837663 20127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:03.838470 20127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:03.839664 20127 hybrid_clock.cc:648] HybridClock initialized: now 1781867523839639 us; error 40 us; skew 500 ppm
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:03.843146 20127 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:03.844233 20127 webserver.cc:460] Webserver started at http://127.7.90.67:33415/ using document root <none> and password file <none>
I20260619 11:12:03.844942 20127 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:03.845037 20127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:03.845315 20127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:03.871299 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "e87129df8b4d427a86caf41da0c64475"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "efdf858de84f2d7e99226d41cca2b0a1"
server_key_iv: "b717994264c5b337f76131f490b3f122"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.872171 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "e87129df8b4d427a86caf41da0c64475"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "efdf858de84f2d7e99226d41cca2b0a1"
server_key_iv: "b717994264c5b337f76131f490b3f122"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.878933 20127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 11:12:03.897891 20142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.899760 20127 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260619 11:12:03.899897 20127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "e87129df8b4d427a86caf41da0c64475"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "efdf858de84f2d7e99226d41cca2b0a1"
server_key_iv: "b717994264c5b337f76131f490b3f122"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:03.900032 20127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:03.924947 20127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:03.928917 20127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:03.929147 20127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:03.929855 20127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:03.930794 20127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:03.930848 20127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.930945 20127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:03.930979 20127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:03.941161 20127 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:45987
I20260619 11:12:03.942332 20127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:03.946450 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20127
I20260619 11:12:03.946590 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:12:03.946939 7529 external_mini_cluster.cc:1428] Setting key c5f5afa7c2650754b308476be6889a8b
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:03.954284 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:45987 every 8 connection(s)
Jun 19 11:12:03 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:03.970956 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.945172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:60591 (local address 127.7.90.126:37811)
0619 11:12:03.945327 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:03.945332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:03.958055 (+ 12723us) server_negotiation.cc:407] Connection header received
0619 11:12:03.959255 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:03.959259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:03.959311 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:03.959386 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:03.960847 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.961618 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.962928 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.963309 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.963426 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.965980 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:03.965984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:03.965986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:03.966012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:03.968402 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.969050 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.969054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.969056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.969146 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:03.969601 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:03.969604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:03.969606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:03.969834 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:03.969904 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:03.970627 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 11:12:03.970765 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:12:03.973042 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:03.945495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.67:60591)
0619 11:12:03.957884 (+ 12389us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:03.957956 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:03.959045 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:03.959585 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:03.959593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:03.959950 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:03.960574 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.960594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.961742 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.961746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.962820 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:03.962834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:03.963509 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:03.963517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:03.963748 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:03.964478 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:03.964494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:03.965815 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:03.968542 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.968548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.968567 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.968930 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.969259 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:03.969262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:03.969264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:03.969455 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:03.969986 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:03.969993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:03.970326 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 11:12:03.971426 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0619 11:12:03.971948 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12090,"thread_start_us":130,"threads_started":1}
I20260619 11:12:03.974208 20257 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:03.974442 20257 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:03.974934 20257 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:03.975893 19772 ts_manager.cc:194] Registered new tserver with Master: e87129df8b4d427a86caf41da0c64475 (127.7.90.67:45987)
I20260619 11:12:03.976554 19772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:60591
I20260619 11:12:03.984548 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:03.985181 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 19853
I20260619 11:12:03.992571 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 19992
I20260619 11:12:04.004709 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20127
I20260619 11:12:04.010692 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 19740
I20260619 11:12:04.017155 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37811
--webserver_interface=127.7.90.126
--webserver_port=34723
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37811
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:04.133440 20261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:04.133728 20261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:04.133805 20261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:04.137730 20261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:04.137776 20261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:04.137794 20261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:04.137809 20261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:04.137864 20261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:04.143041 20261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37811
--ranger_config_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:04.143970 20261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:04.144935 20261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:04.153184 20266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.153235 20267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.153277 20269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:04.153440 20261 server_base.cc:1034] running on GCE node
I20260619 11:12:04.154050 20261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:04.155074 20261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:04.156248 20261 hybrid_clock.cc:648] HybridClock initialized: now 1781867524156235 us; error 25 us; skew 500 ppm
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:04.158900 20261 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:04.159668 20261 webserver.cc:460] Webserver started at http://127.7.90.126:34723/ using document root <none> and password file <none>
I20260619 11:12:04.160168 20261 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:04.160256 20261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:04.204392 20261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:12:04.206993 20277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.208280 20261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:04.208421 20261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "ea75f218761f42c497ebd265ae1f14b9"
format_stamp: "Formatted at 2026-06-19 11:12:02 on dist-test-slave-f1t5"
server_key: "42716b17d5ea7cbfac157505aeec6d74"
server_key_iv: "02886e13d7b1f3612ffc290b1daae7c9"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:04.208947 20261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:04.233141 20261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:04.235529 20261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:04.235721 20261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:04.242718 20261 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37811
I20260619 11:12:04.242745 20329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37811 every 8 connection(s)
I20260619 11:12:04.243705 20261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:04.245997 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20261
I20260619 11:12:04.246425 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:35299
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:37811
--webserver_port=35935
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:04.247831 20330 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:12:04.249310 20330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Bootstrap starting.
I20260619 11:12:04.264927 20330 log.cc:824] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:04.271281 20330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:04.271672 20330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Bootstrap complete.
I20260619 11:12:04.275650 20330 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:04.276034 20330 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea75f218761f42c497ebd265ae1f14b9, State: Initialized, Role: FOLLOWER
I20260619 11:12:04.276479 20330 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:04.276633 20330 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:04.276708 20330 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:04.276821 20330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:12:04.278659 20330 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:04.278973 20330 leader_election.cc:302] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea75f218761f42c497ebd265ae1f14b9; no voters:
I20260619 11:12:04.279278 20330 leader_election.cc:288] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:12:04.279350 20334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:12:04.279582 20334 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [term 2 LEADER]: Becoming Leader. State: Replica: ea75f218761f42c497ebd265ae1f14b9, State: Running, Role: LEADER
I20260619 11:12:04.279891 20334 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } }
I20260619 11:12:04.280375 20330 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:04.281106 20335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea75f218761f42c497ebd265ae1f14b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } } }
I20260619 11:12:04.281229 20335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:04.281541 20336 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea75f218761f42c497ebd265ae1f14b9. Latest consensus state: current_term: 2 leader_uuid: "ea75f218761f42c497ebd265ae1f14b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea75f218761f42c497ebd265ae1f14b9" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37811 } } }
I20260619 11:12:04.281678 20336 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:04.282073 20343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:04.282586 20330 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:12:04.288236 20343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:04.289081 20343 catalog_manager.cc:1259] Loaded cluster ID: 6cc4c83d8a8443419f8f4183261e6b56
I20260619 11:12:04.289172 20343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:04.293445 20343 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:04.294173 20343 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Loaded TSK: 0
I20260619 11:12:04.295392 20343 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:12:04.362499 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:04.362761 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:04.362854 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:04.367267 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:04.367347 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:04.367462 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:04.373219 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:35299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=35935
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:04.374265 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:04.375514 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:04.384905 20367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.385288 20370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:04.388370 20332 server_base.cc:1034] running on GCE node
W20260619 11:12:04.389680 20368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:04.390100 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:04.390738 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:04.391909 20332 hybrid_clock.cc:648] HybridClock initialized: now 1781867524391881 us; error 51 us; skew 500 ppm
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:04.394876 20332 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:04.395829 20332 webserver.cc:460] Webserver started at http://127.7.90.65:35935/ using document root <none> and password file <none>
I20260619 11:12:04.396394 20332 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:04.396500 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:04.427583 20332 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.000s
I20260619 11:12:04.433236 20378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.434752 20332 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260619 11:12:04.434947 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "17c28b4b865248b1ac9007249683f1f9"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "25fbbdef5d1e827764a339a101b3ae96"
server_key_iv: "0e5e7ec63e5d42610384a421b8e60436"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:04.435603 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:04.451108 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:04.454254 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:04.454488 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:04.455183 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:04.456039 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:04.456110 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.456202 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:04.456238 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.467223 20332 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:35299
I20260619 11:12:04.468219 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:04.469216 20491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:35299 every 8 connection(s)
I20260619 11:12:04.477073 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20332
I20260619 11:12:04.477573 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:35231
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:37811
--webserver_port=35463
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:04.490919 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.470724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.65:47149)
0619 11:12:04.471189 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:04.471239 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:04.472050 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:04.474109 (+ 2059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:04.474117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:04.474437 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:04.475060 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.475078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.476997 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.477001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.478429 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.478442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.479513 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.479517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.479690 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.480364 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:04.480380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:04.482327 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:04.487388 (+ 5061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:04.487393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:04.487406 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:04.487735 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:04.488187 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:04.488190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:04.488192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:04.488358 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:04.489131 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:04.489137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:04.489418 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 11:12:04.489653 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 11:12:04.489999 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260619 11:12:04.492375 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.470567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:47149 (local address 127.7.90.126:37811)
0619 11:12:04.472423 (+ 1856us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:04.472439 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:04.472470 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:12:04.472712 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:04.472725 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:04.473156 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:04.473464 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:04.475258 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.476821 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.478586 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.479343 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.479525 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.482645 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:04.482653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:04.482672 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:04.482697 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:04.487249 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.487895 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.487901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.487907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:04.488075 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.488472 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.488475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.488477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:04.488955 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:04.489077 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:04.491438 (+ 2361us) server_negotiation.cc:299] Negotiation successful
0619 11:12:04.491721 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1720,"thread_start_us":109,"threads_started":1}
I20260619 11:12:04.497604 20492 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:04.497887 20492 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.498472 20492 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:04.500607 20492 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:12:04.500761 20492 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.500959 20492 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:04.501868 20492 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.502934 20492 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:04.503839 20492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:12:04.633005 20496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:04.633330 20496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:04.633411 20496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:04.639123 20496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:04.639192 20496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:04.639268 20496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:04.645473 20496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:35231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=35463
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:04.646627 20496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:04.647825 20496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:04.660589 20502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.660589 20501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.661026 20504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:04.661505 20496 server_base.cc:1034] running on GCE node
I20260619 11:12:04.661980 20496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:04.662616 20496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:04.663820 20496 hybrid_clock.cc:648] HybridClock initialized: now 1781867524663785 us; error 50 us; skew 500 ppm
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:04.667003 20496 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:04.667986 20496 webserver.cc:460] Webserver started at http://127.7.90.66:35463/ using document root <none> and password file <none>
I20260619 11:12:04.668529 20496 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:04.668629 20496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:04.686756 20496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:12:04.689472 20512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.691315 20496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:12:04.691457 20496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "65ce7309b5ec46cf93699d439ad7d09c"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "d9eb42e852eed35325d765fab4b9bc59"
server_key_iv: "4cff36f94b9870a1556beeab8396de9a"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:04.692186 20496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:04.707278 20496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:04.710929 20496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:04.711150 20496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:04.711917 20496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:04.713114 20496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:04.713195 20496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.713270 20496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:04.713321 20496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.725080 20496 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35231
I20260619 11:12:04.726689 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35231 every 8 connection(s)
I20260619 11:12:04.727727 20496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:04.735149 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20496
I20260619 11:12:04.735621 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.90.84:38519/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:45987
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:37811
--webserver_port=33415
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:04.751581 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.731351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:47717 (local address 127.7.90.126:37811)
0619 11:12:04.731497 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:04.731502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:04.737613 (+ 6111us) server_negotiation.cc:407] Connection header received
0619 11:12:04.738888 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:04.738893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:04.738954 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:04.739016 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:04.740723 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.741629 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.742925 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.743292 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.743404 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.746796 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:04.746802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:04.746806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:04.746842 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:04.748573 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.749546 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.749549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.749551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:04.749637 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.750040 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.750043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.750045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:04.750290 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:04.750371 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:04.751278 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 11:12:04.751408 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:12:04.753445 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.731837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.66:47717)
0619 11:12:04.737406 (+ 5569us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:04.737464 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:04.738610 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:04.739411 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:04.739420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:04.739773 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:04.740482 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.740501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.741902 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.741904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.742788 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.742796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.744002 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.744007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.744213 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.744930 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:04.744950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:04.746590 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:04.748719 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:04.749038 (+ 319us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:04.749057 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:04.749404 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:04.749747 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:04.749750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:04.749752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:04.749916 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:04.750640 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:04.750648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:04.750991 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 11:12:04.752052 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0619 11:12:04.752384 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5338,"thread_start_us":153,"threads_started":1}
I20260619 11:12:04.755097 20626 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:04.755339 20626 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.755892 20626 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:04.757640 20626 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:12:04.757841 20626 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.758064 20626 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:04.758898 20626 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:04.759121 20626 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:04.759905 20626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:12:04.880419 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:04.880658 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:04.880698 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:04.886379 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:04.886508 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:04.886626 20629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:04.893929 20629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.90.84:38519/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:45987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=33415
--tserver_master_addrs=127.7.90.126:37811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:04.894969 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:04.895987 20629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:12:04.905882 20629 server_base.cc:1034] running on GCE node
W20260619 11:12:04.905926 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.905810 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:04.906069 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:04.906534 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:04.907167 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:04.908370 20629 hybrid_clock.cc:648] HybridClock initialized: now 1781867524908338 us; error 47 us; skew 500 ppm
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:04.911289 20629 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:04.912328 20629 webserver.cc:460] Webserver started at http://127.7.90.67:33415/ using document root <none> and password file <none>
I20260619 11:12:04.912884 20629 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:04.912987 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:04.937424 20629 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:12:04.940598 20646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.942075 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 11:12:04.942209 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "e87129df8b4d427a86caf41da0c64475"
format_stamp: "Formatted at 2026-06-19 11:12:03 on dist-test-slave-f1t5"
server_key: "efdf858de84f2d7e99226d41cca2b0a1"
server_key_iv: "b717994264c5b337f76131f490b3f122"
server_key_version: "kuduclusterkey@0"
I20260619 11:12:04.942785 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:04.964092 20629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:04.967660 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:04.967900 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:04.968605 20629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:04.969691 20629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:04.969795 20629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.969902 20629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:04.969977 20629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:04.980841 20629 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:45987
I20260619 11:12:04.981851 20759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:45987 every 8 connection(s)
I20260619 11:12:04.982420 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:04.987133 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20629
Jun 19 11:12:04 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867524, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:05.003795 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.987897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.7.90.126:37811)
0619 11:12:04.988369 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:04.988373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:04.988385 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:12:04.988431 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:04.988434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:04.988486 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:04.988612 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:04.989448 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.990197 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.991992 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.992340 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.992448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.994163 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:04.994167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:04.994170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:04.994195 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:04.995857 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.996622 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.996628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.996630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:04.996736 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:04.997667 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:04.997672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:04.997673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:05.002629 (+ 4956us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:05.002846 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:05.003462 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 11:12:05.003596 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":83,"threads_started":1}
I20260619 11:12:05.006292 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.985001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:42693 (local address 127.7.90.126:37811)
0619 11:12:04.985134 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:04.985139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:04.987068 (+ 1929us) server_negotiation.cc:407] Connection header received
0619 11:12:04.989236 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:04.989239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:04.989283 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:04.989349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:04.991100 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.991863 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.993597 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.993937 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.994021 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.996796 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:04.996801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:04.996803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:04.996827 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:05.001829 (+ 5002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:05.002922 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:05.002926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:05.002928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:05.003009 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:05.004345 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:05.004349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:05.004351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:05.004560 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:05.004644 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:05.006016 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0619 11:12:05.006131 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:12:05.008944 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:04.985475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37811 (local address 127.7.90.67:42693)
0619 11:12:04.986906 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:04.986964 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:04.988987 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:04.989922 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:04.989932 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:04.990257 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:04.990953 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.990974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.992004 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.992008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.993412 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:04.993423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:04.994206 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:04.994210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:04.994438 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:04.994970 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:04.994991 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:04.996625 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:05.001987 (+ 5362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:05.001995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:05.002012 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:05.002418 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:05.003118 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:05.003123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:05.003125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:05.004209 (+ 1084us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:05.004716 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:05.004725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:05.005087 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 11:12:05.007461 (+ 2374us) client_negotiation.cc:240] Negotiation successful
0619 11:12:05.007824 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":123,"threads_started":1}
I20260619 11:12:05.010473 20760 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37811
I20260619 11:12:05.010710 20760 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:05.011266 20760 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:05.013114 20760 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37811 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:12:05.013240 20760 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:05.013407 20760 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:05.014263 20760 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:05.014420 20760 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
W20260619 11:12:05.015151 20760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 11:12:05 dist-test-slave-f1t5 krb5kdc[19296](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:05.504083 20492 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:05.504379 20492 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:05.505641 20294 ts_manager.cc:194] Registered new tserver with Master: 17c28b4b865248b1ac9007249683f1f9 (127.7.90.65:35299)
I20260619 11:12:05.507311 20294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:47149
Jun 19 11:12:05 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:05.760214 20626 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:05.760533 20626 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:05.761294 20294 ts_manager.cc:194] Registered new tserver with Master: 65ce7309b5ec46cf93699d439ad7d09c (127.7.90.66:35231)
I20260619 11:12:05.762094 20294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:47717
I20260619 11:12:06.015378 20760 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:06.015702 20760 heartbeater.cc:511] Master 127.7.90.126:37811 requested a full tablet report, sending...
I20260619 11:12:06.016458 20294 ts_manager.cc:194] Registered new tserver with Master: e87129df8b4d427a86caf41da0c64475 (127.7.90.67:45987)
I20260619 11:12:06.017186 20294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:42693
I20260619 11:12:06.023094 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:06.038522 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.027372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.7.90.126:37811)
0619 11:12:06.028393 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.028399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.028775 (+ 376us) server_negotiation.cc:407] Connection header received
0619 11:12:06.028820 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.028823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.028872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.028952 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:06.030001 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.030812 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.032047 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.032404 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.032479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.034451 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:06.034455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:06.034458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:06.034486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:06.036367 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:06.036923 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:06.036926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:06.036928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:06.037026 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:06.037401 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:06.037405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:06.037407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:06.037660 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:06.037775 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.038168 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.038328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":92,"threads_started":1}
I20260619 11:12:06.043730 20294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:12:06.048895 20294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 11:12:06 dist-test-slave-f1t5 krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867525, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for HTTP/127.7.90.84@KRBTEST.COM
I20260619 11:12:06.071617 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.062815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35299 (local address 127.0.0.1:59324)
0619 11:12:06.065306 (+ 2491us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.065366 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:06.065485 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:06.066897 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:06.066903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:06.066924 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:06.067041 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.067046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.068371 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.068375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.069779 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.069790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.071077 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.071080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.071186 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.071222 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:12:06.071338 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 11:12:06.071455 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321,"thread_start_us":97,"threads_started":1}
I20260619 11:12:06.072647 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.063513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.7.90.65:35299)
0619 11:12:06.065772 (+ 2259us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.065780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.065802 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:12:06.065874 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.065892 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.066601 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.066731 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:06.067159 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.068235 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.069922 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.070754 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.070874 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.071017 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.072400 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.072509 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2158,"thread_start_us":110,"threads_started":1}
I20260619 11:12:06.076211 20418 tablet_service.cc:1467] Processing CreateTablet for tablet 175a6662781f4db5b8390aa49b4c6b11 (DEFAULT_TABLE table=test-table [id=670a59c866064a31a6dd71873c21054c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:06.077384 20418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175a6662781f4db5b8390aa49b4c6b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:06.078502 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.064807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:45987 (local address 127.0.0.1:51536)
0619 11:12:06.072200 (+ 7393us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.072212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:06.072275 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:06.074387 (+ 2112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:06.074391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:06.074407 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:06.074477 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.074482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.075795 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.075798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.076860 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.076870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.078125 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.078128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.078216 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.078238 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:12:06.078308 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:12:06.078377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7340,"thread_start_us":79,"threads_started":1}
I20260619 11:12:06.079003 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.064981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.7.90.67:45987)
0619 11:12:06.073731 (+ 8750us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.073739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.073765 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:12:06.073861 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.073866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.074078 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.074229 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:06.074603 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.075668 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.076999 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.077806 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.077912 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.078064 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.078759 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.078872 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8626,"thread_start_us":55,"threads_started":1}
I20260619 11:12:06.087690 20694 tablet_service.cc:1467] Processing CreateTablet for tablet 175a6662781f4db5b8390aa49b4c6b11 (DEFAULT_TABLE table=test-table [id=670a59c866064a31a6dd71873c21054c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:06.088788 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175a6662781f4db5b8390aa49b4c6b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:06.102948 20795 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Bootstrap starting.
I20260619 11:12:06.103037 20796 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Bootstrap starting.
I20260619 11:12:06.104452 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.064508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:35231 (local address 127.0.0.1:41154)
0619 11:12:06.066283 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.066295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:06.066385 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:06.099113 (+ 32728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:06.099116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:06.099139 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:06.099252 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.099259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.100844 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.100848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.102107 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.102119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.104021 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.104025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.104123 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.104152 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:12:06.104203 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 11:12:06.104283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"thread_start_us":89,"threads_started":1}
I20260619 11:12:06.105943 20795 tablet_bootstrap.cc:654] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:06.106909 20795 log.cc:824] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:06.107005 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.064681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.7.90.66:35231)
0619 11:12:06.097846 (+ 33165us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.097854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.097881 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:12:06.098578 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.098584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.098833 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.099420 (+ 587us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:06.099570 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.100720 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.102678 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.103528 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.106098 (+ 2570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.106682 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.106747 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.106861 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33041,"thread_start_us":74,"threads_started":1}
I20260619 11:12:06.109282 20796 tablet_bootstrap.cc:654] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:06.109835 20795 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: No bootstrap required, opened a new log
I20260619 11:12:06.110038 20795 ts_tablet_manager.cc:1397] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:12:06.110888 20796 log.cc:824] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:06.111450 20560 tablet_service.cc:1467] Processing CreateTablet for tablet 175a6662781f4db5b8390aa49b4c6b11 (DEFAULT_TABLE table=test-table [id=670a59c866064a31a6dd71873c21054c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:06.112608 20560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175a6662781f4db5b8390aa49b4c6b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:06.115214 20795 raft_consensus.cc:348] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.115607 20795 raft_consensus.cc:374] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:06.115708 20795 raft_consensus.cc:729] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87129df8b4d427a86caf41da0c64475, State: Initialized, Role: FOLLOWER
I20260619 11:12:06.116235 20795 consensus_queue.cc:260] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.117372 20795 ts_tablet_manager.cc:1428] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:12:06.117758 20760 heartbeater.cc:503] Master 127.7.90.126:37811 was elected leader, sending a full tablet report...
I20260619 11:12:06.121148 20796 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: No bootstrap required, opened a new log
I20260619 11:12:06.121397 20796 ts_tablet_manager.cc:1397] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260619 11:12:06.122500 20800 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Bootstrap starting.
I20260619 11:12:06.124876 20800 tablet_bootstrap.cc:654] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:06.125792 20800 log.cc:824] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:06.127903 20800 tablet_bootstrap.cc:492] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: No bootstrap required, opened a new log
I20260619 11:12:06.128089 20800 ts_tablet_manager.cc:1397] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:12:06.132705 20800 raft_consensus.cc:348] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.132994 20800 raft_consensus.cc:374] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:06.133045 20800 raft_consensus.cc:729] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ce7309b5ec46cf93699d439ad7d09c, State: Initialized, Role: FOLLOWER
I20260619 11:12:06.133706 20800 consensus_queue.cc:260] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.134850 20800 ts_tablet_manager.cc:1428] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:12:06.135237 20626 heartbeater.cc:503] Master 127.7.90.126:37811 was elected leader, sending a full tablet report...
I20260619 11:12:06.139259 20796 raft_consensus.cc:348] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.139631 20796 raft_consensus.cc:374] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:06.139744 20796 raft_consensus.cc:729] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c28b4b865248b1ac9007249683f1f9, State: Initialized, Role: FOLLOWER
I20260619 11:12:06.140340 20796 consensus_queue.cc:260] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.141587 20796 ts_tablet_manager.cc:1428] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260619 11:12:06.141978 20492 heartbeater.cc:503] Master 127.7.90.126:37811 was elected leader, sending a full tablet report...
W20260619 11:12:06.221115 20493 tablet.cc:2369] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:06.228657 20799 raft_consensus.cc:482] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:06.228904 20799 raft_consensus.cc:504] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.230381 20799 leader_election.cc:288] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c28b4b865248b1ac9007249683f1f9 (127.7.90.65:35299), 65ce7309b5ec46cf93699d439ad7d09c (127.7.90.66:35231)
W20260619 11:12:06.234459 20761 tablet.cc:2369] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:12:06.235131 20627 tablet.cc:2369] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:06.236338 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.231463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:49655 (local address 127.7.90.65:35299)
0619 11:12:06.231593 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.231599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.231923 (+ 324us) server_negotiation.cc:407] Connection header received
0619 11:12:06.232124 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.232128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.232179 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.232227 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:06.232680 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.233491 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.235060 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.235624 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.235705 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.235773 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.236121 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.236201 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:12:06.236943 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.230609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35299 (local address 127.7.90.67:49655)
0619 11:12:06.231831 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.231846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:06.232003 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:06.232369 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:06.232372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:06.232389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:06.232530 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.232537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.233683 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.233687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.234914 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.234925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.235878 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.235881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.235988 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.236010 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:12:06.236698 (+ 688us) client_negotiation.cc:240] Negotiation successful
0619 11:12:06.236781 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"thread_start_us":127,"threads_started":1}
I20260619 11:12:06.238075 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175a6662781f4db5b8390aa49b4c6b11" candidate_uuid: "e87129df8b4d427a86caf41da0c64475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c28b4b865248b1ac9007249683f1f9" is_pre_election: true
I20260619 11:12:06.238420 20446 raft_consensus.cc:2454] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87129df8b4d427a86caf41da0c64475 in term 0.
I20260619 11:12:06.250912 20650 leader_election.cc:302] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c28b4b865248b1ac9007249683f1f9, e87129df8b4d427a86caf41da0c64475; no voters:
I20260619 11:12:06.251475 20799 raft_consensus.cc:2781] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:06.251585 20799 raft_consensus.cc:482] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:06.251631 20799 raft_consensus.cc:3037] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:06.253068 20799 raft_consensus.cc:504] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.254328 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175a6662781f4db5b8390aa49b4c6b11" candidate_uuid: "e87129df8b4d427a86caf41da0c64475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c28b4b865248b1ac9007249683f1f9"
I20260619 11:12:06.254477 20446 raft_consensus.cc:3037] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:06.255792 20446 raft_consensus.cc:2454] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87129df8b4d427a86caf41da0c64475 in term 1.
I20260619 11:12:06.256261 20650 leader_election.cc:302] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c28b4b865248b1ac9007249683f1f9, e87129df8b4d427a86caf41da0c64475; no voters:
I20260619 11:12:06.253486 20799 leader_election.cc:288] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [CANDIDATE]: Term 1 election: Requested vote from peers 17c28b4b865248b1ac9007249683f1f9 (127.7.90.65:35299), 65ce7309b5ec46cf93699d439ad7d09c (127.7.90.66:35231)
I20260619 11:12:06.256632 20799 raft_consensus.cc:2781] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:06.256701 20799 raft_consensus.cc:686] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 1 LEADER]: Becoming Leader. State: Replica: e87129df8b4d427a86caf41da0c64475, State: Running, Role: LEADER
I20260619 11:12:06.257051 20799 consensus_queue.cc:237] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } }
I20260619 11:12:06.260609 20293 catalog_manager.cc:5495] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 reported cstate change: term changed from 0 to 1, leader changed from <none> to e87129df8b4d427a86caf41da0c64475 (127.7.90.67). New cstate: current_term: 1 leader_uuid: "e87129df8b4d427a86caf41da0c64475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e87129df8b4d427a86caf41da0c64475" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 45987 } health_report { overall_health: HEALTHY } } }
I20260619 11:12:06.264374 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.231021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:35231 (local address 127.7.90.67:57183)
0619 11:12:06.237306 (+ 6285us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.237323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:06.237860 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:06.238221 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:06.238224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:06.238238 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:06.238332 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.238337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.260212 (+ 21875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.260216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.262512 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:06.262527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.263906 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.263909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:06.264021 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.264045 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:12:06.264124 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:12:06.264202 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6215,"thread_start_us":79,"threads_started":1}
I20260619 11:12:06.264418 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.231632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:57183 (local address 127.7.90.66:35231)
0619 11:12:06.231772 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.231776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.237399 (+ 5623us) server_negotiation.cc:407] Connection header received
0619 11:12:06.237994 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.237997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.238050 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.238097 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:06.238445 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.239225 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.262711 (+ 23486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.263362 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.263451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.263518 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.264171 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.264286 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:12:06.265450 20575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175a6662781f4db5b8390aa49b4c6b11" candidate_uuid: "e87129df8b4d427a86caf41da0c64475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ce7309b5ec46cf93699d439ad7d09c" is_pre_election: true
I20260619 11:12:06.265799 20575 raft_consensus.cc:2454] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87129df8b4d427a86caf41da0c64475 in term 0.
I20260619 11:12:06.266137 20576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175a6662781f4db5b8390aa49b4c6b11" candidate_uuid: "e87129df8b4d427a86caf41da0c64475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ce7309b5ec46cf93699d439ad7d09c"
I20260619 11:12:06.266234 20576 raft_consensus.cc:3037] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:06.267519 20576 raft_consensus.cc:2454] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87129df8b4d427a86caf41da0c64475 in term 1.
I20260619 11:12:06.316493 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.312631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.7.90.67:45987)
0619 11:12:06.312762 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:06.312770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:06.312787 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:12:06.312878 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:06.312881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:06.312953 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:06.313082 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:06.313365 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.313991 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.314938 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:06.315179 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:06.315289 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:06.315631 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:06.316078 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:06.316214 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:06.316262 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:12:06.316322 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:12:06.323185 20446 raft_consensus.cc:1261] T 175a6662781f4db5b8390aa49b4c6b11 P 17c28b4b865248b1ac9007249683f1f9 [term 1 FOLLOWER]: Refusing update from remote peer e87129df8b4d427a86caf41da0c64475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:06.323182 20576 raft_consensus.cc:1261] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 1 FOLLOWER]: Refusing update from remote peer e87129df8b4d427a86caf41da0c64475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:06.323995 20806 consensus_queue.cc:1035] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c28b4b865248b1ac9007249683f1f9" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:06.324303 20799 consensus_queue.cc:1035] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ce7309b5ec46cf93699d439ad7d09c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:06.342665 20294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:06.342893 20294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:06.345808 20294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea75f218761f42c497ebd265ae1f14b9: Sending DeleteTablet for 3 replicas of tablet 175a6662781f4db5b8390aa49b4c6b11
I20260619 11:12:06.347005 20560 tablet_service.cc:1514] Processing DeleteTablet for tablet 175a6662781f4db5b8390aa49b4c6b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:06 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:41154
I20260619 11:12:06.347108 20694 tablet_service.cc:1514] Processing DeleteTablet for tablet 175a6662781f4db5b8390aa49b4c6b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:06 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:51536
I20260619 11:12:06.347594 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20332
I20260619 11:12:06.348486 20825 tablet_replica.cc:330] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: stopping tablet replica
I20260619 11:12:06.348897 20825 raft_consensus.cc:2229] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:06.349273 20825 raft_consensus.cc:2258] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:06.353931 20826 tablet_replica.cc:330] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: stopping tablet replica
I20260619 11:12:06.354447 20826 raft_consensus.cc:2229] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:12:06.355002 20826 raft_consensus.cc:2258] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:12:06.355296 20281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:35299 (error 108)
I20260619 11:12:06.355715 20825 ts_tablet_manager.cc:1905] T 175a6662781f4db5b8390aa49b4c6b11 P 65ce7309b5ec46cf93699d439ad7d09c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:06.356581 20826 ts_tablet_manager.cc:1905] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:06.356930 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.356575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:35299 (local address 127.0.0.1:59340)
0619 11:12:06.356763 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.356828 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 11:12:06.357299 20281 catalog_manager.cc:4584] TS 17c28b4b865248b1ac9007249683f1f9 (127.7.90.65:35299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 175a6662781f4db5b8390aa49b4c6b11: Network error: Client connection negotiation failed: client connection to 127.7.90.65:35299: connect: Connection refused (error 111)
I20260619 11:12:06.357750 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20496
I20260619 11:12:06.361133 20826 ts_tablet_manager.cc:1918] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:06.361253 20826 log.cc:1196] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/175a6662781f4db5b8390aa49b4c6b11
I20260619 11:12:06.361646 20826 ts_tablet_manager.cc:1939] T 175a6662781f4db5b8390aa49b4c6b11 P e87129df8b4d427a86caf41da0c64475: Deleting consensus metadata
I20260619 11:12:06.363130 20281 catalog_manager.cc:4841] TS e87129df8b4d427a86caf41da0c64475 (127.7.90.67:45987): tablet 175a6662781f4db5b8390aa49b4c6b11 (table test-table [id=670a59c866064a31a6dd71873c21054c]) successfully deleted
I20260619 11:12:06.364547 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20629
I20260619 11:12:06.364761 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:06.364450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:35231 (local address 127.0.0.1:41158)
0619 11:12:06.364580 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:06.364648 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:35231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 11:12:06.370857 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20261
I20260619 11:12:06.378201 7529 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:12:06.713882 7529 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:12:06.714021 7529 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:12:07.059365 7529 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:12:07.059 UTC [19326] LOG: received smart shutdown request
2026-06-19 11:12:07.063 UTC [19326] LOG: background worker "logical replication launcher" (PID 19335) exited with exit code 1
2026-06-19 11:12:07.063 UTC [19330] LOG: shutting down
2026-06-19 11:12:07.219 UTC [19326] LOG: database system is shut down
2026-06-19T11:12:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46236 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:12:07 dist-test-slave-f1t5 krb5kdc[20836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:12:07 dist-test-slave-f1t5 krb5kdc[20836](info): set up 2 sockets
Jun 19 11:12:07 dist-test-slave-f1t5 krb5kdc[20836](info): commencing operation
krb5kdc: starting...
W20260619 11:12:08.377502 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:12:08.492669 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40509
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:08.600765 20849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:08.601070 20849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:08.601186 20849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:08.605156 20849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:08.605248 20849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:08.605288 20849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:08.605325 20849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:08.605355 20849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:08.610605 20849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:08.611662 20849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:08.612686 20849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:08.620929 20854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:08.620947 20855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:08.620947 20857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:08.621150 20849 server_base.cc:1034] running on GCE node
I20260619 11:12:08.621973 20849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:08.622972 20849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:08.624140 20849 hybrid_clock.cc:648] HybridClock initialized: now 1781867528624129 us; error 28 us; skew 500 ppm
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:08.626947 20849 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:08.627872 20849 webserver.cc:460] Webserver started at http://127.7.90.126:45713/ using document root <none> and password file <none>
I20260619 11:12:08.628376 20849 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:08.628430 20849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:08.628584 20849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:08.630570 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "2b9a0f0067c74deab179aab57beb8962"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "15ba65d1f28271481033f22ffdc907eb"
server_key_iv: "ab09633b5ad637eed37ad4c10d9c54f5"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.631080 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "2b9a0f0067c74deab179aab57beb8962"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "15ba65d1f28271481033f22ffdc907eb"
server_key_iv: "ab09633b5ad637eed37ad4c10d9c54f5"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.634524 20849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:12:08.636735 20864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:08.638028 20849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:08.638156 20849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2b9a0f0067c74deab179aab57beb8962"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "15ba65d1f28271481033f22ffdc907eb"
server_key_iv: "ab09633b5ad637eed37ad4c10d9c54f5"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.638260 20849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:08.652582 20849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:08.655866 20849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:08.656068 20849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:08.663043 20849 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40509
I20260619 11:12:08.663084 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40509 every 8 connection(s)
I20260619 11:12:08.663931 20849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:08.666482 20917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:08.668455 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20849
I20260619 11:12:08.668581 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:12:08.668833 7529 external_mini_cluster.cc:1428] Setting key 3f904ffbd8a85b623a19d805d7e32dc1
I20260619 11:12:08.670188 20917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.90.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.125@KRBTEST.COM" created.
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:08.685864 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.672274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40509 (local address 127.0.0.1:36320)
0619 11:12:08.672792 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.672847 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:08.674131 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:08.674797 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:08.674802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:08.675257 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:08.675682 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.675704 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.676946 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.676949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.677779 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.677789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.678396 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.678399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.678491 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.679124 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:08.679151 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:08.680847 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:08.682836 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.682848 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.682866 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.683188 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.683491 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.683496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.683500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.683664 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.684238 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:08.684247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:08.684429 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 11:12:08.684653 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 11:12:08.685056 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":63,"thread_start_us":138,"threads_started":1}
I20260619 11:12:08.685863 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.672274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.7.90.126:40509)
0619 11:12:08.672769 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:08.672786 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:08.672974 (+ 188us) server_negotiation.cc:407] Connection header received
0619 11:12:08.674428 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:08.674432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:08.674596 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:08.674740 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:08.675877 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.676824 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.677890 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.678249 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.678371 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.680993 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:08.681002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:08.681016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:08.681043 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:08.682730 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.683291 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.683296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.683297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.683394 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.683764 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.683766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.683767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.684118 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:08.684218 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:08.684747 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 11:12:08.684979 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":383,"spinlock_wait_cycles":46848,"thread_start_us":141,"threads_started":1}
I20260619 11:12:08.688326 20917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:12:08.689085 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.688836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57942)
0619 11:12:08.688953 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.689030 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 11:12:08.689347 20865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
I20260619 11:12:08.690246 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.690027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57944)
0619 11:12:08.690159 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.690200 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 11:12:08.690414 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:32819: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
I20260619 11:12:08.694242 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:32819
--webserver_interface=127.7.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:08.739653 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } attempt: 1
I20260619 11:12:08.740610 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.740287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57960)
0619 11:12:08.740459 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.740533 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 11:12:08.741348 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.741143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57964)
0619 11:12:08.741248 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.741298 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:12:08.741469 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:32819: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
W20260619 11:12:08.805051 20924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:08.805394 20924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:08.805478 20924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:08.809464 20924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:08.809585 20924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:08.809638 20924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:08.809680 20924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:08.809708 20924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:08.809669 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } attempt: 2
I20260619 11:12:08.810578 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.810213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57966)
0619 11:12:08.810390 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.810461 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 11:12:08.811297 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.811077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57968)
0619 11:12:08.811186 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.811233 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:12:08.811424 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:32819: Network error: Client connection negotiation failed: client connection to 127.7.90.125:32819: connect: Connection refused (error 111)
I20260619 11:12:08.815387 20924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:08.816444 20924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:08.817672 20924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:08.825838 20929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:08.825814 20930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:08.825846 20932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:08.826216 20924 server_base.cc:1034] running on GCE node
I20260619 11:12:08.826943 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:08.828118 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:08.829322 20924 hybrid_clock.cc:648] HybridClock initialized: now 1781867528829305 us; error 42 us; skew 500 ppm
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:08.832252 20924 init.cc:377] Logged in from keytab as kudu/127.7.90.125@KRBTEST.COM (short username kudu)
I20260619 11:12:08.833097 20924 webserver.cc:460] Webserver started at http://127.7.90.125:36879/ using document root <none> and password file <none>
I20260619 11:12:08.833675 20924 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:08.833740 20924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:08.833906 20924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:08.835824 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data/instance:
uuid: "495aeebd37f1486ea2ca5872b6a355ac"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "a5504cb67ce92ea1fb1b097029ac0950"
server_key_iv: "3e5dcf86b8e46f978a2bfbe743977a72"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.836333 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal/instance:
uuid: "495aeebd37f1486ea2ca5872b6a355ac"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "a5504cb67ce92ea1fb1b097029ac0950"
server_key_iv: "3e5dcf86b8e46f978a2bfbe743977a72"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.839811 20924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:12:08.842391 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:08.843600 20924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:12:08.843695 20924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "495aeebd37f1486ea2ca5872b6a355ac"
format_stamp: "Formatted at 2026-06-19 11:12:08 on dist-test-slave-f1t5"
server_key: "a5504cb67ce92ea1fb1b097029ac0950"
server_key_iv: "3e5dcf86b8e46f978a2bfbe743977a72"
server_key_version: "encryptionkey@0"
I20260619 11:12:08.843776 20924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:08.857409 20924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:08.860639 20924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:08.860783 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:08.867666 20924 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.125:32819
I20260619 11:12:08.867727 20991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.125:32819 every 8 connection(s)
I20260619 11:12:08.868629 20924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
I20260619 11:12:08.870662 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20924
I20260619 11:12:08.870744 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-1/wal/instance
I20260619 11:12:08.870997 7529 external_mini_cluster.cc:1428] Setting key 8f7a669c56c3048bd131235a0386237a
I20260619 11:12:08.871344 20992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:08.876627 20992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.124@KRBTEST.COM" created.
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:08.890954 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.878961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.7.90.126:40509)
0619 11:12:08.879135 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:08.879140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:08.879922 (+ 782us) server_negotiation.cc:407] Connection header received
0619 11:12:08.880906 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:08.880909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:08.880989 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:08.881089 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:08.882416 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.883202 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.884311 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.884555 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.884667 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.887313 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:08.887316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:08.887318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:08.887350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:08.888823 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.889342 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.889347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.889354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.889425 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.889788 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.889793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.889795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.889978 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:08.890084 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:08.890613 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 11:12:08.890739 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:12:08.891693 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.879294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40509 (local address 127.0.0.1:36336)
0619 11:12:08.879774 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.879821 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:08.880711 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:08.881241 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:08.881247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:08.881753 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:08.882275 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.882289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.883345 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.883349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.884187 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.884196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.884701 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.884705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.884871 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.885469 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:08.885484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:08.887137 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:08.888948 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.888954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.888968 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.889229 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.889534 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.889540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.889545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.889690 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.890081 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:08.890088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:08.890362 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 11:12:08.890544 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 11:12:08.890869 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":166,"threads_started":1}
I20260619 11:12:08.892993 20992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
I20260619 11:12:08.895771 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:44533
--webserver_interface=127.7.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:12:08.902609 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } attempt: 3
I20260619 11:12:08.902930 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.893482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57972)
0619 11:12:08.893645 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.893660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:08.893736 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:08.894344 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:08.894347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:08.894539 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:08.894634 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.894640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.895788 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.895794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.896721 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.896735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.897312 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.897315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.897426 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.897948 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:08.897959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:08.899353 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:08.901268 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.901273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.901275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.901471 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.901893 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.901899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.901901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.901983 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.902420 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:08.902423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:08.902504 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 11:12:08.902622 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:12:08.902750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 11:12:08.903239 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.893665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.7.90.125:32819)
0619 11:12:08.894019 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:08.894029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:08.894056 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:12:08.894088 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:08.894092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:08.894233 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:08.894338 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:08.894729 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.895665 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.896846 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.897196 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.897317 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.899473 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:08.899488 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:08.899499 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:08.899517 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:08.901129 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.901701 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.901707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.901711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.901791 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.902072 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.902075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.902077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.902310 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:08.902457 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:08.902787 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 11:12:08.902963 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":158,"threads_started":1}
I20260619 11:12:08.905500 20992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:12:08.906311 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.905977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56904)
0619 11:12:08.906126 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.906223 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 11:12:08.907486 20942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
I20260619 11:12:08.908619 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.908388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56912)
0619 11:12:08.908491 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.908544 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:12:08.908890 20992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Jun 19 11:12:08 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:12:08.912808 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.903645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.7.90.125:32819)
0619 11:12:08.904002 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:08.904008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:08.904018 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:12:08.904045 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:08.904047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:08.904078 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:08.904178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:08.904955 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.906043 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.906921 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.907312 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.907441 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.909816 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:08.909819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:08.909822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:08.909875 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:08.911330 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.911774 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.911776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.911777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.911864 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:08.912129 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:08.912131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:08.912132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:08.912287 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:08.912361 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:08.912607 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 11:12:08.912696 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
I20260619 11:12:08.912935 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.903085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57980)
0619 11:12:08.903670 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.903687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:08.903775 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:08.904207 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:08.904211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:08.904388 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:08.904833 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.904840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.906134 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.906138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.906785 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:08.906794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:08.907435 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:08.907440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:08.907564 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:08.908059 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:08.908073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:08.909611 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:08.911426 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.911430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.911432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.911680 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.911947 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:08.911950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:08.911952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:08.912048 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:08.912368 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:08.912371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:08.912469 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 11:12:08.912572 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:12:08.912712 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498}
I20260619 11:12:08.913702 20917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:12:08.914376 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.914142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56916)
0619 11:12:08.914255 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.914309 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 11:12:08.915036 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.914792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56930)
0619 11:12:08.914937 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.914977 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 11:12:08.915203 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
I20260619 11:12:08.946382 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 1
I20260619 11:12:08.947319 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.946967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56936)
0619 11:12:08.947135 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.947236 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 11:12:08.948087 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.947880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56942)
0619 11:12:08.947984 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.948036 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:12:08.948210 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
I20260619 11:12:08.958141 20992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 1
I20260619 11:12:08.958905 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.958632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56944)
0619 11:12:08.958762 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.958817 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 11:12:08.959604 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:08.959389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56958)
0619 11:12:08.959497 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:08.959549 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:12:08.959718 20992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
W20260619 11:12:09.008775 20998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:09.009007 20998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:09.009042 20998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:09.013046 20998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:09.013093 20998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:09.013113 20998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:09.013128 20998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:09.013182 20998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:09.019236 20998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40509,127.7.90.125:32819,127.7.90.124:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:09.020248 20998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:09.021232 20998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:12:09.023370 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 2
I20260619 11:12:09.024092 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.023819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56974)
0619 11:12:09.023957 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.024023 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:12:09.024806 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.024590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56978)
0619 11:12:09.024690 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.024738 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 11:12:09.024935 20917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
I20260619 11:12:09.027871 20992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 2
I20260619 11:12:09.028533 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.028266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56982)
0619 11:12:09.028413 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.028466 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 11:12:09.029431 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.029159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:56994)
0619 11:12:09.029274 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.029327 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 11:12:09.029665 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:09.029767 20992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:44533: Network error: Client connection negotiation failed: client connection to 127.7.90.124:44533: connect: Connection refused (error 111)
W20260619 11:12:09.029661 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:09.029901 20998 server_base.cc:1034] running on GCE node
W20260619 11:12:09.029683 21008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:09.030503 20998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:09.031498 20998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:09.032665 20998 hybrid_clock.cc:648] HybridClock initialized: now 1781867529032654 us; error 20 us; skew 500 ppm
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:09.035290 20998 init.cc:377] Logged in from keytab as kudu/127.7.90.124@KRBTEST.COM (short username kudu)
I20260619 11:12:09.036185 20998 webserver.cc:460] Webserver started at http://127.7.90.124:33773/ using document root <none> and password file <none>
I20260619 11:12:09.036713 20998 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:09.036772 20998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:09.036993 20998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:09.038995 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data/instance:
uuid: "75445c0a67b44d96b4bf630469b930d7"
format_stamp: "Formatted at 2026-06-19 11:12:09 on dist-test-slave-f1t5"
server_key: "6a8ab43a3da29d885283603a3a0f7f71"
server_key_iv: "9c5f1eeda4f591734e65f254405c92a0"
server_key_version: "encryptionkey@0"
I20260619 11:12:09.039564 20998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal/instance:
uuid: "75445c0a67b44d96b4bf630469b930d7"
format_stamp: "Formatted at 2026-06-19 11:12:09 on dist-test-slave-f1t5"
server_key: "6a8ab43a3da29d885283603a3a0f7f71"
server_key_iv: "9c5f1eeda4f591734e65f254405c92a0"
server_key_version: "encryptionkey@0"
I20260619 11:12:09.043211 20998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:12:09.045631 21015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:09.046890 20998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:12:09.047052 20998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "75445c0a67b44d96b4bf630469b930d7"
format_stamp: "Formatted at 2026-06-19 11:12:09 on dist-test-slave-f1t5"
server_key: "6a8ab43a3da29d885283603a3a0f7f71"
server_key_iv: "9c5f1eeda4f591734e65f254405c92a0"
server_key_version: "encryptionkey@0"
I20260619 11:12:09.047170 20998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:09.079914 20998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:09.083087 20998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:09.083299 20998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:09.090692 20998 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.124:44533
I20260619 11:12:09.090708 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.124:44533 every 8 connection(s)
I20260619 11:12:09.091632 20998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
I20260619 11:12:09.092403 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 20998
I20260619 11:12:09.092497 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867375844525-7529-0/minicluster-data/master-2/wal/instance
I20260619 11:12:09.092762 7529 external_mini_cluster.cc:1428] Setting key 40a09e101788b7a278a94a101025555b
I20260619 11:12:09.094808 21068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:09.099277 21068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:09.108657 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.094176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.7.90.124:44533)
0619 11:12:09.094776 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.094796 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.094861 (+ 65us) server_negotiation.cc:407] Connection header received
0619 11:12:09.095679 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.095700 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.096109 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.096468 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.097155 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.098669 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.099474 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.100130 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.100291 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.103534 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.103547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.103556 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.103582 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.105472 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.106012 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.106017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.106023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.106125 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.106424 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.106428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.106433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.106881 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.107077 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.107305 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.107636 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":129,"threads_started":1}
I20260619 11:12:09.110463 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.101204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.7.90.126:40509)
0619 11:12:09.101364 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.101371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.101710 (+ 339us) server_negotiation.cc:407] Connection header received
0619 11:12:09.101884 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.101888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.101939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.102026 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.102842 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.103419 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.104307 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.104514 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.104626 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.106852 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.106857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.106860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.106884 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.108680 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.109174 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.109178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.109179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.109263 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.109558 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.109566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.109567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.109848 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.109937 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.110262 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.110358 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:12:09.110553 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.101149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40509 (local address 127.0.0.1:36352)
0619 11:12:09.101600 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.101637 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:09.101765 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:09.102079 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:09.102084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:09.102542 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:09.102710 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.102718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.103546 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.103552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.104185 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.104193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.104635 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.104638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.104772 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.105213 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:09.105224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:09.106719 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:09.108825 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.108835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.108842 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.109074 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.109360 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.109363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.109364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.109447 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.109956 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:09.109962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:09.110072 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 11:12:09.110213 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 11:12:09.110379 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":166,"threads_started":1}
I20260619 11:12:09.111578 21068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:09.119866 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.112065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:32819 (local address 127.0.0.1:57988)
0619 11:12:09.112161 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.112176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:09.112243 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:09.112486 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:09.112489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:09.112646 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:09.112727 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.112732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.113488 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.113491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.114028 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.114035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.114512 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.114515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.114608 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.115135 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:09.115146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:09.116393 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:09.118429 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.118431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.118433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.118607 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.118894 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.118897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.118899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.118990 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.119421 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:09.119423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:09.119496 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 11:12:09.119576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:12:09.119700 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 11:12:09.120177 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.112157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.7.90.125:32819)
0619 11:12:09.112278 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.112284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.112299 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:12:09.112340 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.112342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.112381 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.112474 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.112869 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.113374 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.114143 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.114401 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.114513 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.116522 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.116526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.116528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.116561 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.118324 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.118731 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.118734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.118735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.118804 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.119088 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.119091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.119092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.119327 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.119423 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.119656 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.119782 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:12:09.120322 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.111565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.7.90.126:40509)
0619 11:12:09.111663 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.111669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.111741 (+ 72us) server_negotiation.cc:407] Connection header received
0619 11:12:09.111891 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.111893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.111953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.112010 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.112747 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.113217 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.114070 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.114278 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.114384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.117053 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.117056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.117059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.117107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.118576 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.118988 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.118990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.118991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.119081 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.119349 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.119355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.119360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.119582 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.119820 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.119927 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.120033 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:12:09.120911 20992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 3
I20260619 11:12:09.121420 21068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:12:09.124069 20917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } attempt: 3
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:12:09 dist-test-slave-f1t5 krb5kdc[20836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867528, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:12:09.132388 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.121281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:57012)
0619 11:12:09.121414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.121430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:09.121499 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:09.121922 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:09.121925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:09.122103 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:09.122204 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.122209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.123629 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.123636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.124444 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.124453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.125340 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.125346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.125737 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.126552 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:09.126569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:09.127892 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:09.129866 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.129870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.129872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.130139 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.130795 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.130801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.130803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.130896 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.131619 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:09.131625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:09.131956 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 11:12:09.132056 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 11:12:09.132187 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:12:09.132702 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.121956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.7.90.124:44533)
0619 11:12:09.123768 (+ 1812us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.123776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.123785 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:12:09.123816 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.123817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.123846 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.123944 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.124335 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.125022 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.125970 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.126328 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.126453 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.129221 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.129225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.129230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.129259 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.131007 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131445 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.131448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.131451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.131542 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131811 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.131814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.131816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.131991 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.132061 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.132367 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.132489 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":1742,"thread_start_us":126,"threads_started":1}
I20260619 11:12:09.132994 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.121322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.7.90.124:44533)
0619 11:12:09.121667 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.121671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.121684 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:12:09.121717 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.121722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.121763 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.122312 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.122391 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.123501 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.124596 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.124931 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.125023 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.128045 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.128051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.128057 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.128083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.129763 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.130366 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.130373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.130375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.130459 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131000 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.131003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.131005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.131476 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.131572 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.132728 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.132849 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260619 11:12:09.132802 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.121869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:57016)
0619 11:12:09.121997 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.122011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:09.122080 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:09.123952 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:09.123955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:09.124131 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:09.124203 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.124209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.125118 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.125124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.125843 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.125850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.126454 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.126458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.126613 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.127012 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:09.127026 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:09.129059 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:09.131118 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.131127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.131351 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.131644 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.131653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.131724 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.132120 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:09.132123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:09.132215 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:12:09.132430 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 11:12:09.132550 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260619 11:12:09.133288 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.121815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.7.90.125:32819)
0619 11:12:09.121914 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.121918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.122019 (+ 101us) server_negotiation.cc:407] Connection header received
0619 11:12:09.122193 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.122196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.122235 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.122321 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.122840 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.123564 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.124582 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.124903 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.124999 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.129457 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.129461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.129465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.129493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.131377 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.131825 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.131829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.131832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.131917 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.132226 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.132230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.132231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.132425 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.132621 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.132962 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.133144 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:12:09.135001 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.124416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:44533 (local address 127.0.0.1:57024)
0619 11:12:09.124615 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:09.124627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:09.124692 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:09.125645 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:09.125650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:09.125823 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:09.125930 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.125939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.126768 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.126772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.127602 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:09.127613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.128090 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.128094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:09.128204 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.128769 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:09.128788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:09.130694 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:09.133381 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.133384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.133386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.133685 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.134029 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:09.134032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:09.134034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:09.134130 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:09.134537 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:09.134540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:09.134628 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 11:12:09.134713 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:12:09.134855 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 11:12:09.135571 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.125134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.7.90.124:44533)
0619 11:12:09.125389 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.125397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.125408 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:12:09.125436 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.125439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.125472 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.125617 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.126028 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.126638 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.127727 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.127994 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.128085 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.130846 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.130852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.130854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.130875 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.133261 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.133836 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.133838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.133840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.133920 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.134241 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.134244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.134246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.134424 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.135242 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.135312 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.135413 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
I20260619 11:12:09.136687 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: Bootstrap starting.
I20260619 11:12:09.136691 20992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: Bootstrap starting.
I20260619 11:12:09.138115 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: Bootstrap starting.
I20260619 11:12:09.139290 21068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:09.139381 20992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:09.139987 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:12:09.140120 21068 log.cc:824] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:09.140120 20992 log.cc:824] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:09.140333 20917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:09.140924 20917 log.cc:824] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:09.142501 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: No bootstrap required, opened a new log
I20260619 11:12:09.142518 20992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: No bootstrap required, opened a new log
I20260619 11:12:09.142917 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: No bootstrap required, opened a new log
I20260619 11:12:09.146394 20992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.146481 20917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.146672 20917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:09.146725 20917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9a0f0067c74deab179aab57beb8962, State: Initialized, Role: FOLLOWER
I20260619 11:12:09.146819 20992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:09.147267 20992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495aeebd37f1486ea2ca5872b6a355ac, State: Initialized, Role: FOLLOWER
I20260619 11:12:09.147504 20917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.147583 21068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.147853 21068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:09.147922 21068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75445c0a67b44d96b4bf630469b930d7, State: Initialized, Role: FOLLOWER
I20260619 11:12:09.148051 20992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.148348 21068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.149046 20917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:09.149623 21068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:09.150230 20992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:09.152207 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.142484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.7.90.124:44533)
0619 11:12:09.142587 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.142591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.142809 (+ 218us) server_negotiation.cc:407] Connection header received
0619 11:12:09.143001 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.143004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.143050 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.143097 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.144072 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.144820 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.145972 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.146312 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.146400 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.148251 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.148254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.148257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.148279 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.149963 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.150513 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.150531 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.150535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.150643 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.150999 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.151002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.151003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.151221 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.151345 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.151883 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.152013 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 11:12:09.156904 21104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 11:12:09.158505 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.158842 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.159011 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.159145 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 11:12:09.160709 21120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:12:09.160840 21104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 11:12:09.157573 20870 tablet.cc:2369] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:09.161303 20998 master_runner.cc:263] No leader master found from master 75445c0a67b44d96b4bf630469b930d7
I20260619 11:12:09.161363 20998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:12:09.159438 20849 master_runner.cc:263] No leader master found from master 2b9a0f0067c74deab179aab57beb8962
I20260619 11:12:09.161464 20849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:12:09.157138 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.161670 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.161814 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:09.162000 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.164363 21041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b9a0f0067c74deab179aab57beb8962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75445c0a67b44d96b4bf630469b930d7" is_pre_election: true
I20260619 11:12:09.164629 21041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9a0f0067c74deab179aab57beb8962 in term 0.
I20260619 11:12:09.165396 20867 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9a0f0067c74deab179aab57beb8962, 75445c0a67b44d96b4bf630469b930d7; no voters:
I20260619 11:12:09.165997 21083 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 495aeebd37f1486ea2ca5872b6a355ac (127.7.90.125:32819), 75445c0a67b44d96b4bf630469b930d7 (127.7.90.124:44533)
I20260619 11:12:09.166146 21083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:09.166253 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:09.166354 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 0 FOLLOWER]: Advancing to term 1
W20260619 11:12:09.168498 21115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:12:09.168579 21115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:12:09.162533 20924 master_runner.cc:263] No leader master found from master 495aeebd37f1486ea2ca5872b6a355ac
I20260619 11:12:09.168792 20924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:12:09.163763 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b9a0f0067c74deab179aab57beb8962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495aeebd37f1486ea2ca5872b6a355ac" is_pre_election: true
I20260619 11:12:09.169018 20966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b9a0f0067c74deab179aab57beb8962 in term 0.
I20260619 11:12:09.169111 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.147204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.7.90.126:40509)
0619 11:12:09.147316 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.147320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.152778 (+ 5458us) server_negotiation.cc:407] Connection header received
0619 11:12:09.154596 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.154599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.154655 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.154718 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.160278 (+ 5560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.161023 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.162375 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.162719 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.162797 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.164641 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.164645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.164648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.164674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.167219 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.167732 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.167736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.167739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.167824 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.168147 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.168151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.168153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.168352 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.168460 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.168829 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.168950 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:12:09.169801 21120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:12:09.170089 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.170778 20966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b9a0f0067c74deab179aab57beb8962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495aeebd37f1486ea2ca5872b6a355ac"
I20260619 11:12:09.170894 20966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:09.172693 20966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9a0f0067c74deab179aab57beb8962 in term 1.
I20260619 11:12:09.173049 20865 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b9a0f0067c74deab179aab57beb8962, 495aeebd37f1486ea2ca5872b6a355ac; no voters:
I20260619 11:12:09.173270 21084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:09.173588 21084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9a0f0067c74deab179aab57beb8962, State: Running, Role: LEADER
I20260619 11:12:09.173972 21084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } }
I20260619 11:12:09.175215 21041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b9a0f0067c74deab179aab57beb8962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75445c0a67b44d96b4bf630469b930d7"
I20260619 11:12:09.175343 21041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:09.177153 21041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b9a0f0067c74deab179aab57beb8962 in term 1.
I20260619 11:12:09.178828 21084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9a0f0067c74deab179aab57beb8962. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.179067 21084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:09.178996 21083 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [CANDIDATE]: Term 1 election: Requested vote from peers 495aeebd37f1486ea2ca5872b6a355ac (127.7.90.125:32819), 75445c0a67b44d96b4bf630469b930d7 (127.7.90.124:44533)
I20260619 11:12:09.179497 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:09.149369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.7.90.125:32819)
0619 11:12:09.149471 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:09.149475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:09.169698 (+ 20223us) server_negotiation.cc:407] Connection header received
0619 11:12:09.171030 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:09.171033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:09.171083 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:09.171136 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:09.171979 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.172785 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.173966 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:09.174283 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:09.174367 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:09.176002 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:09.176006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:09.176009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:09.176036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:09.177693 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.178199 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.178203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.178204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.178282 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:09.178583 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:09.178587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:09.178589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:09.178787 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:09.178880 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:09.179228 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:12:09.179340 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:12:09.181808 21126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:09.186141 21126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:09.197324 21041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [term 1 FOLLOWER]: Refusing update from remote peer 2b9a0f0067c74deab179aab57beb8962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:09.198052 21083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:09.198612 20966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [term 1 FOLLOWER]: Refusing update from remote peer 2b9a0f0067c74deab179aab57beb8962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:09.199802 21083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:09.202729 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9a0f0067c74deab179aab57beb8962. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.202862 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.203711 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9a0f0067c74deab179aab57beb8962. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.203824 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.214004 21084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.214125 21084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:09.214215 21127 mvcc.cc:205] Tried to move back new op lower bound from 7298529399583866880 to 7298529399512457216. Current Snapshot: MvccSnapshot[applied={T|T < 7298529399583866880}]
I20260619 11:12:09.214359 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.214473 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.214730 21130 mvcc.cc:205] Tried to move back new op lower bound from 7298529399583866880 to 7298529399512457216. Current Snapshot: MvccSnapshot[applied={T|T < 7298529399583866880}]
I20260619 11:12:09.216681 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.216807 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:09.219750 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b9a0f0067c74deab179aab57beb8962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b9a0f0067c74deab179aab57beb8962" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40509 } } peers { permanent_uuid: "495aeebd37f1486ea2ca5872b6a355ac" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 32819 } } peers { permanent_uuid: "75445c0a67b44d96b4bf630469b930d7" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 44533 } } }
I20260619 11:12:09.219851 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:12:09.220266 21128 mvcc.cc:205] Tried to move back new op lower bound from 7298529399583866880 to 7298529399512457216. Current Snapshot: MvccSnapshot[applied={T|T < 7298529399583866880}]
I20260619 11:12:09.223009 21126 catalog_manager.cc:1347] Generated new cluster ID: bd59e7964cc34670b2c4efc8c449234c
I20260619 11:12:09.223063 21126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:09.242254 21126 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:12:09.243523 21126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:09.263926 21126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b9a0f0067c74deab179aab57beb8962: Generated new TSK 0
I20260619 11:12:09.264978 21126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:12:10.169947 21115 catalog_manager.cc:1259] Loaded cluster ID: bd59e7964cc34670b2c4efc8c449234c
I20260619 11:12:10.170045 21115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: loading cluster ID for follower catalog manager: success
I20260619 11:12:10.171034 21120 catalog_manager.cc:1259] Loaded cluster ID: bd59e7964cc34670b2c4efc8c449234c
I20260619 11:12:10.171099 21120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: loading cluster ID for follower catalog manager: success
I20260619 11:12:10.172271 21115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: acquiring CA information for follower catalog manager: success
I20260619 11:12:10.173012 21120 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: acquiring CA information for follower catalog manager: success
I20260619 11:12:10.173712 21115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75445c0a67b44d96b4bf630469b930d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:12:10.174158 21120 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 495aeebd37f1486ea2ca5872b6a355ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:12:10.335310 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20849
I20260619 11:12:10.341473 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20924
I20260619 11:12:10.347189 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 20998
2026-06-19T11:12:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3014 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:12:10 dist-test-slave-f1t5 krb5kdc[21143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:12:10 dist-test-slave-f1t5 krb5kdc[21143](info): set up 2 sockets
Jun 19 11:12:10 dist-test-slave-f1t5 krb5kdc[21143](info): commencing operation
krb5kdc: starting...
W20260619 11:12:11.389959 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:12:11.504632 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37975
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:11.614306 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:11.614545 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:11.614585 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:11.618607 21156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:11.618654 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:11.618671 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:11.618711 21156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:11.618741 21156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:11.623739 21156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:11.624665 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:11.625648 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:11.633865 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:11.633859 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:11.633859 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:11.633937 21156 server_base.cc:1034] running on GCE node
I20260619 11:12:11.634644 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:11.635681 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:11.636853 21156 hybrid_clock.cc:648] HybridClock initialized: now 1781867531636843 us; error 28 us; skew 500 ppm
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867531, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:11.639498 21156 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:11.640383 21156 webserver.cc:460] Webserver started at http://127.7.90.126:45059/ using document root <none> and password file <none>
I20260619 11:12:11.641131 21156 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:11.641252 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:11.641486 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:11.643476 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "6bd5b407f2154dafaf89c56aaf775980"
format_stamp: "Formatted at 2026-06-19 11:12:11 on dist-test-slave-f1t5"
server_key: "ef7b8492401831c7b8a8296800583b74"
server_key_iv: "fd467be21c6866a429e01b7cdcf6d549"
server_key_version: "encryptionkey@0"
I20260619 11:12:11.644039 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "6bd5b407f2154dafaf89c56aaf775980"
format_stamp: "Formatted at 2026-06-19 11:12:11 on dist-test-slave-f1t5"
server_key: "ef7b8492401831c7b8a8296800583b74"
server_key_iv: "fd467be21c6866a429e01b7cdcf6d549"
server_key_version: "encryptionkey@0"
I20260619 11:12:11.647580 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:12:11.649742 21171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:11.650875 21156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:12:11.651043 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "6bd5b407f2154dafaf89c56aaf775980"
format_stamp: "Formatted at 2026-06-19 11:12:11 on dist-test-slave-f1t5"
server_key: "ef7b8492401831c7b8a8296800583b74"
server_key_iv: "fd467be21c6866a429e01b7cdcf6d549"
server_key_version: "encryptionkey@0"
I20260619 11:12:11.651151 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:11.670370 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:11.673511 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:11.673759 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:11.680142 21156 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37975
I20260619 11:12:11.680153 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37975 every 8 connection(s)
I20260619 11:12:11.681058 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:11.683599 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:11.688889 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980: Bootstrap starting.
I20260619 11:12:11.690621 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21156
I20260619 11:12:11.690706 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:12:11.690974 7529 external_mini_cluster.cc:1428] Setting key c551aeb86a321bed928203422a72115e
I20260619 11:12:11.691385 21224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:11.692236 21224 log.cc:824] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:11.694245 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980: No bootstrap required, opened a new log
I20260619 11:12:11.698271 21224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } }
I20260619 11:12:11.698549 21224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:11.698625 21224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd5b407f2154dafaf89c56aaf775980, State: Initialized, Role: FOLLOWER
I20260619 11:12:11.699142 21224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } }
I20260619 11:12:11.699322 21224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:11.699390 21224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:11.699486 21224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:11.700362 21224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } }
I20260619 11:12:11.700678 21224 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd5b407f2154dafaf89c56aaf775980; no voters:
I20260619 11:12:11.701028 21224 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:12:11.701119 21229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:11.701336 21229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd5b407f2154dafaf89c56aaf775980, State: Running, Role: LEADER
I20260619 11:12:11.701678 21229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } }
I20260619 11:12:11.702327 21224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:11.702724 21229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd5b407f2154dafaf89c56aaf775980. Latest consensus state: current_term: 1 leader_uuid: "6bd5b407f2154dafaf89c56aaf775980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } } }
I20260619 11:12:11.702847 21229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:11.703128 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd5b407f2154dafaf89c56aaf775980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd5b407f2154dafaf89c56aaf775980" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37975 } } }
I20260619 11:12:11.703217 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:11.703328 21233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:11.706625 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:11.692033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.7.90.126:37975)
0619 11:12:11.692455 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:11.692484 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:11.692518 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:12:11.693266 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:11.693284 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:11.693674 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:11.693942 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:11.694843 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:11.695797 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:11.697023 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:11.697460 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:11.697660 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:11.700413 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:11.700419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:11.700450 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:11.700483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:11.702598 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:11.703558 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:11.703564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:11.703570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:11.703705 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:11.704038 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:11.704044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:11.704048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:11.704519 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:11.704692 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:11.705427 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 11:12:11.705727 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":147,"threads_started":1}
I20260619 11:12:11.706673 21233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:11.709465 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:12:11.712862 21233 catalog_manager.cc:1347] Generated new cluster ID: 0a079cae33d2455b9fe76b6ef0997ae1
I20260619 11:12:11.712924 21233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:11.725122 21233 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:12:11.726104 21233 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:11.731361 21233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd5b407f2154dafaf89c56aaf775980: Generated new TSK 0
I20260619 11:12:11.732074 21233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:12:11.765369 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21156
2026-06-19T11:12:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1416 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21251](info): set up 2 sockets
Jun 19 11:12:11 dist-test-slave-f1t5 krb5kdc[21251](info): commencing operation
krb5kdc: starting...
W20260619 11:12:12.789651 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:12:12 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:12:12.902341 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:13.006933 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:13.007161 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:13.007198 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:13.011065 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:13.011112 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:13.011132 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:13.011147 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:13.011162 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:13.016229 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:13.017175 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:13.018119 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:13.026196 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:13.026286 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.026252 21264 server_base.cc:1034] running on GCE node
W20260619 11:12:13.026196 21272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.026911 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:13.027886 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:13.029035 21264 hybrid_clock.cc:648] HybridClock initialized: now 1781867533029012 us; error 43 us; skew 500 ppm
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:13.034966 21264 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:13.035867 21264 webserver.cc:460] Webserver started at http://127.7.90.126:38689/ using document root <none> and password file <none>
I20260619 11:12:13.036417 21264 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:13.036505 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:13.036736 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:13.038789 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "23837494044e4974a5b3fcd6852367ed"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "5313bc5fa509942971ee3d1afc25f0e0"
server_key_iv: "eab857c6bf61b33776a26d466797cc0b"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.039330 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "23837494044e4974a5b3fcd6852367ed"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "5313bc5fa509942971ee3d1afc25f0e0"
server_key_iv: "eab857c6bf61b33776a26d466797cc0b"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.042827 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:12:13.045058 21279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.046237 21264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:12:13.046375 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "23837494044e4974a5b3fcd6852367ed"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "5313bc5fa509942971ee3d1afc25f0e0"
server_key_iv: "eab857c6bf61b33776a26d466797cc0b"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.046479 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:13.061470 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:13.064287 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:13.064500 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:13.071554 21264 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37119
I20260619 11:12:13.071832 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37119 every 8 connection(s)
I20260619 11:12:13.072525 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:13.075166 21332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:13.077725 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21264
I20260619 11:12:13.077852 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:12:13.078145 7529 external_mini_cluster.cc:1428] Setting key 793996758f23be035bc41730d60fdaca
I20260619 11:12:13.081010 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap starting.
I20260619 11:12:13.083519 21332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:13.084375 21332 log.cc:824] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:13.086457 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: No bootstrap required, opened a new log
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:13.091198 21332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:13.091490 21332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:13.091598 21332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Initialized, Role: FOLLOWER
I20260619 11:12:13.092082 21332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:13.092236 21332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:13.092322 21332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:13.092437 21332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:13.092573 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.079468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.7.90.126:37119)
0619 11:12:13.079877 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.079892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.079926 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:12:13.080527 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.080554 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.080860 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.081160 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:13.082000 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.082965 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.084133 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.084655 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.084830 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.087570 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:13.087576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:13.087592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:13.087621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:13.089652 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.090047 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.090053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.090059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.090199 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.090529 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.090531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.090533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.090909 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:13.091065 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.091361 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.091700 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260619 11:12:13.093379 21332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:13.093762 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23837494044e4974a5b3fcd6852367ed; no voters:
I20260619 11:12:13.094092 21332 leader_election.cc:288] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:12:13.094188 21337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:13.094379 21337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 LEADER]: Becoming Leader. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Running, Role: LEADER
I20260619 11:12:13.094743 21337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:13.095360 21332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:13.096184 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:13.096310 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:13.096678 21346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:13.096639 21339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23837494044e4974a5b3fcd6852367ed. Latest consensus state: current_term: 1 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:13.096827 21339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:13.099877 21346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:13.105432 21346 catalog_manager.cc:1347] Generated new cluster ID: 90b7bf96ff014d3db375db2f1058e30f
I20260619 11:12:13.105495 21346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:13.116860 21346 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:12:13.118121 21346 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:13.129951 21346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Generated new TSK 0
I20260619 11:12:13.130785 21346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:12:13.159034 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:13.283939 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:13.284268 21358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:13.284317 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:13.288453 21358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:13.288512 21358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:13.288568 21358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:13.293716 21358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:13.294734 21358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:13.295717 21358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:13.304824 21363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:13.304826 21366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:13.304908 21364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.305259 21358 server_base.cc:1034] running on GCE node
I20260619 11:12:13.305640 21358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:13.306185 21358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:13.307353 21358 hybrid_clock.cc:648] HybridClock initialized: now 1781867533307313 us; error 51 us; skew 500 ppm
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:13.310026 21358 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:13.310883 21358 webserver.cc:460] Webserver started at http://127.7.90.65:37325/ using document root <none> and password file <none>
I20260619 11:12:13.311426 21358 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:13.311486 21358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:13.311728 21358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:13.313808 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "198b8769b39b441bac62077a8f5d742d"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "e75177bd5905b0c5e847f617b67b0f55"
server_key_iv: "139fcec1aeb24bfbd24ff45b3f3279b0"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.314388 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "198b8769b39b441bac62077a8f5d742d"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "e75177bd5905b0c5e847f617b67b0f55"
server_key_iv: "139fcec1aeb24bfbd24ff45b3f3279b0"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.318037 21358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:12:13.320464 21373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.321785 21358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:13.321908 21358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "198b8769b39b441bac62077a8f5d742d"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "e75177bd5905b0c5e847f617b67b0f55"
server_key_iv: "139fcec1aeb24bfbd24ff45b3f3279b0"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.322058 21358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:13.333166 21358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:13.335963 21358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:13.336190 21358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:13.336766 21358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:13.337586 21358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:13.337630 21358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.337702 21358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:13.337729 21358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.346403 21358 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36907
I20260619 11:12:13.346427 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36907 every 8 connection(s)
I20260619 11:12:13.347317 21358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:13.355443 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21358
I20260619 11:12:13.355589 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:12:13.355921 7529 external_mini_cluster.cc:1428] Setting key cd7b5d97732f9aefc26ddc3d9c51257f
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:13.361624 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.349030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:42419 (local address 127.7.90.126:37119)
0619 11:12:13.349184 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.349188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.350117 (+ 929us) server_negotiation.cc:407] Connection header received
0619 11:12:13.351456 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.351460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.351530 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.351629 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:13.352758 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.353326 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.354560 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.354802 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.354908 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.357733 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:13.357736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:13.357738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:13.357760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:13.359466 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.359955 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.359961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.359964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.360059 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.360439 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.360441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.360442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.360611 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:13.360684 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.361303 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.361402 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 11:12:13.362736 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.349478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:42419)
0619 11:12:13.349964 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.350015 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.351242 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.351801 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.351807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.352152 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:13.352606 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.352618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.353454 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.353464 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.354411 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.354420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.354941 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.354945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.355113 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.355735 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:13.355757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:13.357581 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:13.359596 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.359601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.359612 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.359846 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.360160 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.360166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.360169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.360343 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.360716 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:13.360725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:13.361054 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.361255 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.361649 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":99,"threads_started":1}
I20260619 11:12:13.365185 21487 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:13.365432 21487 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:12:13.365980 21487 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:13.367440 21296 ts_manager.cc:194] Registered new tserver with Master: 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
I20260619 11:12:13.368999 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:42419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:12:13.382618 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:13.487859 21492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:13.488094 21492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:13.488133 21492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:13.492132 21492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:13.492192 21492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:13.492247 21492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:13.497650 21492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:13.498644 21492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:13.499598 21492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:13.508873 21497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:13.508863 21498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.509018 21492 server_base.cc:1034] running on GCE node
W20260619 11:12:13.509229 21500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.509689 21492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:13.510298 21492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:13.511477 21492 hybrid_clock.cc:648] HybridClock initialized: now 1781867533511464 us; error 24 us; skew 500 ppm
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:13.514173 21492 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:13.514917 21492 webserver.cc:460] Webserver started at http://127.7.90.66:39243/ using document root <none> and password file <none>
I20260619 11:12:13.515409 21492 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:13.515465 21492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:13.515620 21492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:13.517465 21492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "340b9e8d27bc14e76442dc5b34c4ffae"
server_key_iv: "f5b2ebc0d8fe1d930649e9dcef9790c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.518055 21492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "340b9e8d27bc14e76442dc5b34c4ffae"
server_key_iv: "f5b2ebc0d8fe1d930649e9dcef9790c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.521499 21492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:12:13.523901 21507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.525056 21492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:12:13.525187 21492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "340b9e8d27bc14e76442dc5b34c4ffae"
server_key_iv: "f5b2ebc0d8fe1d930649e9dcef9790c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.525281 21492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:13.535458 21492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:13.538527 21492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:13.538719 21492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:13.539260 21492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:13.540015 21492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:13.540059 21492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.540123 21492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:13.540164 21492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.548753 21492 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:38955
I20260619 11:12:13.548772 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:38955 every 8 connection(s)
I20260619 11:12:13.549644 21492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:13.558646 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21492
I20260619 11:12:13.558784 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:12:13.559101 7529 external_mini_cluster.cc:1428] Setting key 1e21b4a70d963ecd4e68f6711eeed584
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:13.563802 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.551476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:38791 (local address 127.7.90.126:37119)
0619 11:12:13.551621 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.551624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.552432 (+ 808us) server_negotiation.cc:407] Connection header received
0619 11:12:13.553760 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.553763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.553820 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.553923 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:13.555067 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.555767 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.557190 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.557404 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.557482 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.559974 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:13.559977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:13.559979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:13.560001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:13.561720 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.562222 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.562228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.562233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.562329 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.562744 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.562747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.562748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.562914 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:13.562991 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.563593 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.563687 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:12:13.564954 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.551858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:38791)
0619 11:12:13.552286 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.552338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.553491 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.554099 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.554106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.554445 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:13.554919 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.554932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.555901 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.555908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.557080 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.557089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.557568 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.557573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.557720 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.558277 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:13.558292 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:13.559802 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:13.561823 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.561827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.561837 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.562125 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.562441 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.562446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.562448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.562647 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.563014 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:13.563020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:13.563329 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.563569 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.563904 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260619 11:12:13.566177 21621 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:13.566439 21621 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:13.566939 21621 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:13.567993 21296 ts_manager.cc:194] Registered new tserver with Master: 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
I20260619 11:12:13.568819 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:38791
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:12:13.584530 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:13.693298 21626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:13.693627 21626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:13.693675 21626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:13.697706 21626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:13.697750 21626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:13.697801 21626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:13.702541 21626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:13.703392 21626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:13.704255 21626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:13.713792 21631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:13.713807 21632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.713840 21626 server_base.cc:1034] running on GCE node
W20260619 11:12:13.713825 21634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:13.714447 21626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:13.715052 21626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:13.716229 21626 hybrid_clock.cc:648] HybridClock initialized: now 1781867533716214 us; error 26 us; skew 500 ppm
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:13.718819 21626 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:13.719712 21626 webserver.cc:460] Webserver started at http://127.7.90.67:35297/ using document root <none> and password file <none>
I20260619 11:12:13.720255 21626 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:13.720342 21626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:13.720556 21626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:13.723043 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "153cd8013dac4f9f832e587533c2de2b"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "6b8c6e4f0df1478ff0794bdf59080d43"
server_key_iv: "de2b864f0c70b594e21b885914c6d1c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.723596 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "153cd8013dac4f9f832e587533c2de2b"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "6b8c6e4f0df1478ff0794bdf59080d43"
server_key_iv: "de2b864f0c70b594e21b885914c6d1c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.727254 21626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:12:13.729502 21641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.730729 21626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:13.730825 21626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "153cd8013dac4f9f832e587533c2de2b"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "6b8c6e4f0df1478ff0794bdf59080d43"
server_key_iv: "de2b864f0c70b594e21b885914c6d1c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:13.730899 21626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:13.752998 21626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:13.756387 21626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:13.756537 21626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:13.757057 21626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:13.757804 21626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:13.757848 21626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.757886 21626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:13.757906 21626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:13.766909 21626 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:35451
I20260619 11:12:13.766937 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:35451 every 8 connection(s)
I20260619 11:12:13.767792 21626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:13.770401 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21626
I20260619 11:12:13.770526 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:12:13.770817 7529 external_mini_cluster.cc:1428] Setting key 41a6446527db6da5da5361f573222769
Jun 19 11:12:13 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867533, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:13.780773 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.769573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:60331 (local address 127.7.90.126:37119)
0619 11:12:13.769716 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.769721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.770608 (+ 887us) server_negotiation.cc:407] Connection header received
0619 11:12:13.772298 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.772301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.772364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.772438 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:13.773782 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.774302 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.775384 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.775592 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.775664 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.777970 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:13.777973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:13.777975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:13.777996 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:13.779171 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.779600 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.779603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.779604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.779678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.779951 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.779953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.779954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.780101 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:13.780171 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.780575 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.780661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:12:13.781560 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.769954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:60331)
0619 11:12:13.770442 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.770506 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.772040 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.772582 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.772589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.773080 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:13.773657 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.773674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.774410 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.774413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.775285 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.775294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.775709 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.775712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.775849 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.776443 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:13.776459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:13.777831 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:13.779265 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.779269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.779279 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.779510 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.779751 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:13.779753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:13.779755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:13.779871 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:13.780181 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:13.780186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:13.780385 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.780523 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.780739 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260619 11:12:13.782541 21755 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:13.782763 21755 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:13.783226 21755 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:13.784168 21296 ts_manager.cc:194] Registered new tserver with Master: 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451)
I20260619 11:12:13.784613 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:13.784812 21296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:60331
I20260619 11:12:13.793453 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.786679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.7.90.126:37119)
0619 11:12:13.786819 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.786823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.786972 (+ 149us) server_negotiation.cc:407] Connection header received
0619 11:12:13.787125 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.787130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.787170 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.787258 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:13.787913 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.788388 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.789213 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.789439 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.789583 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.790729 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:13.790733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:13.790735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:13.790758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:13.791893 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.792288 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.792292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.792294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.792370 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:13.792633 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:13.792638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:13.792640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:13.792830 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:13.792963 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.793195 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.793297 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:12:13.796043 21296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:12:13.799100 21296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:12:13.812767 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.807073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.0.0.1:35014)
0619 11:12:13.807693 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.807707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.807791 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.808783 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.808786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.808803 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:13.808891 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.808897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.809957 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.809962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.811080 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.811089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812276 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.812376 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812398 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.812468 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.812622 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":23,"thread_start_us":117,"threads_started":1}
I20260619 11:12:13.812779 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.807849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.7.90.67:35451)
0619 11:12:13.808546 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.808557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.808572 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:12:13.808626 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.808630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.808817 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.808990 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:13.809184 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.810203 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.811571 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812114 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812241 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812387 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.812544 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.812638 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":143,"threads_started":1}
I20260619 11:12:13.812927 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.806881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.0.0.1:36344)
0619 11:12:13.807194 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.807247 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.807372 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.808455 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.808461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.808483 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:13.808604 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.808610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.810023 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.810027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.811257 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.811280 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812551 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.812645 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812665 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.812734 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.812799 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":6,"thread_start_us":56,"threads_started":1}
I20260619 11:12:13.812974 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.807972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.7.90.65:36907)
0619 11:12:13.808362 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.808373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.808395 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:12:13.808464 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.808468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.808655 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.808798 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:13.809046 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.809840 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.811228 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812131 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812584 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812715 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.812763 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.812842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":107,"threads_started":1}
I20260619 11:12:13.813417 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.807291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.7.90.66:38955)
0619 11:12:13.807773 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.807784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.807806 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:12:13.807887 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.807894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.808115 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.808262 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:13.808731 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.809881 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.811408 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812252 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812477 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:13.813159 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 11:12:13.813267 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":110,"threads_started":1}
I20260619 11:12:13.813853 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.807551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:35451 (local address 127.0.0.1:47566)
0619 11:12:13.808298 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.808309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.808374 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.808988 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.808991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.809011 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:13.809094 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.809100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.810342 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.810345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.811453 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.811460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.812239 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:13.812243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:13.812345 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:13.812385 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 11:12:13.812496 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:12:13.813717 (+ 1221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260619 11:12:13.816009 21555 tablet_service.cc:1467] Processing CreateTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 (DEFAULT_TABLE table=test-table [id=108adec386ca41848efb084aaa6e2b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:13.816164 21689 tablet_service.cc:1467] Processing CreateTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 (DEFAULT_TABLE table=test-table [id=108adec386ca41848efb084aaa6e2b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:13.816166 21421 tablet_service.cc:1467] Processing CreateTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 (DEFAULT_TABLE table=test-table [id=108adec386ca41848efb084aaa6e2b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:13.817028 21555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8a0c2e89254ef384b25be4a1cabb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:13.817135 21689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8a0c2e89254ef384b25be4a1cabb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:13.817128 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8a0c2e89254ef384b25be4a1cabb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:13.823726 21772 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Bootstrap starting.
I20260619 11:12:13.824398 21774 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Bootstrap starting.
I20260619 11:12:13.824785 21773 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Bootstrap starting.
I20260619 11:12:13.826478 21772 tablet_bootstrap.cc:654] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:13.826886 21773 tablet_bootstrap.cc:654] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:13.827348 21772 log.cc:824] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:13.828020 21773 log.cc:824] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:13.829385 21772 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: No bootstrap required, opened a new log
I20260619 11:12:13.829615 21772 ts_tablet_manager.cc:1397] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:12:13.830024 21773 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: No bootstrap required, opened a new log
I20260619 11:12:13.830153 21774 tablet_bootstrap.cc:654] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:13.830229 21773 ts_tablet_manager.cc:1397] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:12:13.830935 21774 log.cc:824] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:13.832855 21774 tablet_bootstrap.cc:492] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: No bootstrap required, opened a new log
I20260619 11:12:13.832808 21772 raft_consensus.cc:348] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.833019 21774 ts_tablet_manager.cc:1397] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 11:12:13.833091 21772 raft_consensus.cc:374] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:13.833160 21772 raft_consensus.cc:729] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732a1b42d98b4029820d19bc4f3cfbb1, State: Initialized, Role: FOLLOWER
I20260619 11:12:13.833693 21772 consensus_queue.cc:260] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.833920 21773 raft_consensus.cc:348] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.834261 21773 raft_consensus.cc:374] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:13.834322 21773 raft_consensus.cc:729] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198b8769b39b441bac62077a8f5d742d, State: Initialized, Role: FOLLOWER
I20260619 11:12:13.834342 21621 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
I20260619 11:12:13.834712 21773 consensus_queue.cc:260] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.835678 21773 ts_tablet_manager.cc:1428] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:12:13.835685 21772 ts_tablet_manager.cc:1428] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:12:13.836079 21487 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
I20260619 11:12:13.836599 21774 raft_consensus.cc:348] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.836763 21774 raft_consensus.cc:374] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:13.836892 21774 raft_consensus.cc:729] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153cd8013dac4f9f832e587533c2de2b, State: Initialized, Role: FOLLOWER
I20260619 11:12:13.837324 21774 consensus_queue.cc:260] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.838195 21774 ts_tablet_manager.cc:1428] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:12:13.838222 21755 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
W20260619 11:12:13.848388 21488 tablet.cc:2369] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:13.996848 21780 raft_consensus.cc:482] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:13.997133 21780 raft_consensus.cc:504] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:13.998378 21780 leader_election.cc:288] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955), 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907)
I20260619 11:12:14.001931 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.998579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.7.90.67:54533)
0619 11:12:13.998709 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.998725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.998815 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.999147 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.999150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.999194 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:13.999310 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.999315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.000023 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.000030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.000824 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.000833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.001473 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.001476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.001623 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.001641 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.001732 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.001802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"spinlock_wait_cycles":34432}
I20260619 11:12:14.001930 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.998765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:54533 (local address 127.7.90.66:38955)
0619 11:12:13.998900 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.998905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.998919 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:12:13.998978 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.998987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.999036 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.999134 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:13.999421 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:13.999914 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.000949 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.001354 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.001471 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.001534 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.001746 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.001813 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:12:14.002224 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.998692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.7.90.67:59441)
0619 11:12:13.999058 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:13.999068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:13.999204 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:13.999484 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:13.999495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:13.999509 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:13.999636 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:13.999641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.000333 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.000338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.001098 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.001105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.001799 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.001802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.001899 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.001927 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.002009 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.002077 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"spinlock_wait_cycles":4352,"thread_start_us":138,"threads_started":1}
I20260619 11:12:14.002215 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:13.998809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:59441 (local address 127.7.90.65:36907)
0619 11:12:13.998935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:13.998939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:13.999161 (+ 222us) server_negotiation.cc:407] Connection header received
0619 11:12:13.999325 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:13.999328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:13.999379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:13.999476 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:13.999751 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.000225 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.001226 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.001683 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.001797 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.001854 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.002046 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.002115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:12:14.002652 21575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8a0c2e89254ef384b25be4a1cabb04" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" is_pre_election: true
I20260619 11:12:14.002983 21575 raft_consensus.cc:2454] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 0.
I20260619 11:12:14.002971 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8a0c2e89254ef384b25be4a1cabb04" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198b8769b39b441bac62077a8f5d742d" is_pre_election: true
I20260619 11:12:14.003225 21441 raft_consensus.cc:2454] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 0.
I20260619 11:12:14.003465 21643 leader_election.cc:302] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1; no voters:
I20260619 11:12:14.003731 21780 raft_consensus.cc:2781] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:14.003814 21780 raft_consensus.cc:482] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:14.003857 21780 raft_consensus.cc:3037] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.004835 21780 raft_consensus.cc:504] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:14.005167 21780 leader_election.cc:288] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 election: Requested vote from peers 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955), 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907)
I20260619 11:12:14.005481 21575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8a0c2e89254ef384b25be4a1cabb04" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
I20260619 11:12:14.005647 21575 raft_consensus.cc:3037] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.005702 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8a0c2e89254ef384b25be4a1cabb04" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198b8769b39b441bac62077a8f5d742d"
I20260619 11:12:14.005827 21441 raft_consensus.cc:3037] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.006688 21575 raft_consensus.cc:2454] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 1.
I20260619 11:12:14.006862 21441 raft_consensus.cc:2454] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 1.
I20260619 11:12:14.006942 21643 leader_election.cc:302] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1; no voters:
I20260619 11:12:14.007128 21780 raft_consensus.cc:2781] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:14.007375 21780 raft_consensus.cc:686] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 1 LEADER]: Becoming Leader. State: Replica: 153cd8013dac4f9f832e587533c2de2b, State: Running, Role: LEADER
I20260619 11:12:14.007697 21780 consensus_queue.cc:237] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } }
I20260619 11:12:14.010777 21296 catalog_manager.cc:5495] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 153cd8013dac4f9f832e587533c2de2b (127.7.90.67). New cstate: current_term: 1 leader_uuid: "153cd8013dac4f9f832e587533c2de2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } health_report { overall_health: HEALTHY } } }
W20260619 11:12:14.018666 21756 tablet.cc:2369] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:14.046113 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.042799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.7.90.67:35451)
0619 11:12:14.042921 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.042925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.042938 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:12:14.043094 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.043096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.043152 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.043236 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:14.043439 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.043940 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.044749 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.044953 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.045029 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.045272 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:14.045739 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:14.045831 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.045888 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.045934 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 11:12:14.050869 21622 tablet.cc:2369] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:14.051996 21575 raft_consensus.cc:1261] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Refusing update from remote peer 153cd8013dac4f9f832e587533c2de2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:14.052086 21441 raft_consensus.cc:1261] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Refusing update from remote peer 153cd8013dac4f9f832e587533c2de2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:14.052793 21782 consensus_queue.cc:1035] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:14.052994 21780 consensus_queue.cc:1035] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:14.061914 21786 mvcc.cc:205] Tried to move back new op lower bound from 7298529419469242368 to 7298529419297935360. Current Snapshot: MvccSnapshot[applied={T|T < 7298529419469242368}]
I20260619 11:12:14.062223 21787 mvcc.cc:205] Tried to move back new op lower bound from 7298529419469242368 to 7298529419297935360. Current Snapshot: MvccSnapshot[applied={T|T < 7298529419469242368}]
I20260619 11:12:14.069186 21296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:14.069321 21296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:14.071743 21296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Sending DeleteTablet for 3 replicas of tablet de8a0c2e89254ef384b25be4a1cabb04
I20260619 11:12:14.072615 21689 tablet_service.cc:1514] Processing DeleteTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:14 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:47566
I20260619 11:12:14.072633 21421 tablet_service.cc:1514] Processing DeleteTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:14 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:35014
I20260619 11:12:14.073069 21795 tablet_replica.cc:330] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: stopping tablet replica
I20260619 11:12:14.073360 21795 raft_consensus.cc:2229] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:14.073709 21795 raft_consensus.cc:2258] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:14.075009 21795 ts_tablet_manager.cc:1905] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:14.077687 21795 ts_tablet_manager.cc:1918] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:14.077725 21794 tablet_replica.cc:330] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: stopping tablet replica
I20260619 11:12:14.077872 21795 log.cc:1196] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/de8a0c2e89254ef384b25be4a1cabb04
I20260619 11:12:14.077823 21555 tablet_service.cc:1514] Processing DeleteTablet for tablet de8a0c2e89254ef384b25be4a1cabb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:14 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:36344
I20260619 11:12:14.077976 21794 raft_consensus.cc:2229] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:12:14.078327 21795 ts_tablet_manager.cc:1939] T de8a0c2e89254ef384b25be4a1cabb04 P 198b8769b39b441bac62077a8f5d742d: Deleting consensus metadata
I20260619 11:12:14.078342 21794 raft_consensus.cc:2258] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:14.078694 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21358
I20260619 11:12:14.080789 21794 ts_tablet_manager.cc:1905] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:14.083272 21794 ts_tablet_manager.cc:1918] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:14.083359 21794 log.cc:1196] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/de8a0c2e89254ef384b25be4a1cabb04
I20260619 11:12:14.083781 21794 ts_tablet_manager.cc:1939] T de8a0c2e89254ef384b25be4a1cabb04 P 153cd8013dac4f9f832e587533c2de2b: Deleting consensus metadata
I20260619 11:12:14.084930 21280 catalog_manager.cc:4841] TS 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): tablet de8a0c2e89254ef384b25be4a1cabb04 (table test-table [id=108adec386ca41848efb084aaa6e2b60]) successfully deleted
I20260619 11:12:14.085336 21796 tablet_replica.cc:330] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: stopping tablet replica
I20260619 11:12:14.085724 21796 raft_consensus.cc:2229] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:14.086059 21796 raft_consensus.cc:2258] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:14.087301 21796 ts_tablet_manager.cc:1905] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:12:14.089263 21280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:36907 (error 108)
I20260619 11:12:14.089890 21796 ts_tablet_manager.cc:1918] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:14.090044 21796 log.cc:1196] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/de8a0c2e89254ef384b25be4a1cabb04
I20260619 11:12:14.090071 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21492
I20260619 11:12:14.090401 21796 ts_tablet_manager.cc:1939] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting consensus metadata
I20260619 11:12:14.092665 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.092305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.0.0.1:35022)
0619 11:12:14.092526 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.092593 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 11:12:14.092803 21280 catalog_manager.cc:4584] TS 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de8a0c2e89254ef384b25be4a1cabb04: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36907: connect: Connection refused (error 111)
I20260619 11:12:14.097908 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21626
I20260619 11:12:14.099351 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.098006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.0.0.1:36356)
0619 11:12:14.099257 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.099298 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:38955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1187}
I20260619 11:12:14.103437 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21264
I20260619 11:12:14.109566 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--webserver_interface=127.7.90.126
--webserver_port=38689
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:14.214598 21798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:14.214818 21798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:14.214856 21798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:14.218757 21798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:14.218804 21798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:14.218822 21798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:14.218837 21798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:14.218851 21798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:14.223999 21798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:14.224949 21798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:14.225967 21798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:14.238305 21807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.238340 21805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.238307 21804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:14.238523 21798 server_base.cc:1034] running on GCE node
I20260619 11:12:14.239190 21798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:14.240259 21798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:14.241436 21798 hybrid_clock.cc:648] HybridClock initialized: now 1781867534241399 us; error 49 us; skew 500 ppm
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:14.243985 21798 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:14.244905 21798 webserver.cc:460] Webserver started at http://127.7.90.126:38689/ using document root <none> and password file <none>
I20260619 11:12:14.245478 21798 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:14.245609 21798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:14.249086 21798 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:12:14.251082 21814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.252213 21798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:14.252345 21798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "23837494044e4974a5b3fcd6852367ed"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "5313bc5fa509942971ee3d1afc25f0e0"
server_key_iv: "eab857c6bf61b33776a26d466797cc0b"
server_key_version: "encryptionkey@0"
I20260619 11:12:14.252924 21798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:14.267426 21798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:14.269965 21798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:14.270154 21798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:14.276791 21798 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37119
I20260619 11:12:14.277050 21866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37119 every 8 connection(s)
I20260619 11:12:14.277819 21798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:14.281790 21867 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:12:14.283303 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap starting.
I20260619 11:12:14.285272 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21798
I20260619 11:12:14.285737 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:36907
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=37325
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:14.300419 21867 log.cc:824] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:14.308426 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:14.308759 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap complete.
I20260619 11:12:14.312598 21867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:14.312978 21867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Initialized, Role: FOLLOWER
I20260619 11:12:14.313421 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:14.313612 21867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:14.313697 21867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:14.313814 21867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:12:14.315541 21867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:14.315855 21867 leader_election.cc:302] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23837494044e4974a5b3fcd6852367ed; no voters:
I20260619 11:12:14.316159 21867 leader_election.cc:288] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:12:14.316236 21871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:12:14.316422 21871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 LEADER]: Becoming Leader. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Running, Role: LEADER
I20260619 11:12:14.316694 21871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:14.317307 21867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:14.317561 21872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:14.317596 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23837494044e4974a5b3fcd6852367ed. Latest consensus state: current_term: 2 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:14.317703 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:14.317690 21872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:14.318068 21875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:14.322544 21875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:14.323288 21875 catalog_manager.cc:1259] Loaded cluster ID: 90b7bf96ff014d3db375db2f1058e30f
I20260619 11:12:14.323347 21875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:14.324698 21875 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 11:12:14.329597 21875 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 11:12:14.334533 21875 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:14.335193 21875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Loaded TSK: 0
I20260619 11:12:14.336361 21875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:12:14.396880 21869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:14.397125 21869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:14.397187 21869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:14.401031 21869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:14.401096 21869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:14.401181 21869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:14.406183 21869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:36907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=37325
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:14.407132 21869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:14.408100 21869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:14.417217 21895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.417284 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.417263 21897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:14.418200 21869 server_base.cc:1034] running on GCE node
I20260619 11:12:14.418649 21869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:14.419269 21869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:14.420469 21869 hybrid_clock.cc:648] HybridClock initialized: now 1781867534420443 us; error 41 us; skew 500 ppm
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:14.423307 21869 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:14.424242 21869 webserver.cc:460] Webserver started at http://127.7.90.65:37325/ using document root <none> and password file <none>
I20260619 11:12:14.424805 21869 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:14.424909 21869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:14.428758 21869 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:14.430881 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.432144 21869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:14.432273 21869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "198b8769b39b441bac62077a8f5d742d"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "e75177bd5905b0c5e847f617b67b0f55"
server_key_iv: "139fcec1aeb24bfbd24ff45b3f3279b0"
server_key_version: "encryptionkey@0"
I20260619 11:12:14.432821 21869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:14.449579 21869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:14.452766 21869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:14.452965 21869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:14.453610 21869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:14.454457 21869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:14.454531 21869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.454602 21869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:14.454653 21869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.464389 21869 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36907
I20260619 11:12:14.464457 22017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36907 every 8 connection(s)
I20260619 11:12:14.465492 21869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:14.472203 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 21869
I20260619 11:12:14.472649 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:38955
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=39243
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:14.481812 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.467429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:39899 (local address 127.7.90.126:37119)
0619 11:12:14.467759 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.467772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.468331 (+ 559us) server_negotiation.cc:407] Connection header received
0619 11:12:14.469733 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.469744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.470041 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.470305 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:14.471275 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.472426 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.473696 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.474255 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.474380 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.477139 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:14.477144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:14.477155 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:14.477192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:14.478816 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.479359 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.479364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.479368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.479472 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.479801 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.479805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.479807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.480097 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:14.480229 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.480719 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.480949 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260619 11:12:14.481824 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.467741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:39899)
0619 11:12:14.468173 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.468212 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.469307 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.470384 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.470390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.470718 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:14.471137 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.471150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.472551 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.472554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.473491 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.473501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.474426 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.474433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.474647 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.475247 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:14.475271 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:14.476952 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:14.478964 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.478969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.478980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.479224 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.479567 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.479572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.479573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.479713 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.480205 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:14.480210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:14.480428 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.480613 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.480921 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260619 11:12:14.483928 22018 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:14.484163 22018 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:14.484668 22018 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:14.486232 21831 ts_manager.cc:194] Registered new tserver with Master: 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907)
I20260619 11:12:14.487370 21831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:39899
W20260619 11:12:14.580695 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:14.580911 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:14.580966 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:14.585013 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:14.585094 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:14.585194 22022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:14.590117 22022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=39243
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:14.591004 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:14.591902 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:14.600680 22027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.600704 22028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:14.600745 22022 server_base.cc:1034] running on GCE node
W20260619 11:12:14.600680 22030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:14.601239 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:14.601855 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:14.603031 22022 hybrid_clock.cc:648] HybridClock initialized: now 1781867534603019 us; error 24 us; skew 500 ppm
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:14.605585 22022 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:14.606339 22022 webserver.cc:460] Webserver started at http://127.7.90.66:39243/ using document root <none> and password file <none>
I20260619 11:12:14.606837 22022 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:14.606935 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:14.610258 22022 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:14.612128 22037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.613222 22022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:14.613350 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "340b9e8d27bc14e76442dc5b34c4ffae"
server_key_iv: "f5b2ebc0d8fe1d930649e9dcef9790c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:14.613916 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:14.634629 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:14.637375 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:14.637611 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:14.638139 22022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:14.639086 22044 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 11:12:14.642432 22044 ts_tablet_manager.cc:1852] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 11:12:14.642791 22044 ts_tablet_manager.cc:1905] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:14.646252 22044 ts_tablet_manager.cc:1918] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:14.646438 22044 ts_tablet_manager.cc:1939] T de8a0c2e89254ef384b25be4a1cabb04 P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting consensus metadata
I20260619 11:12:14.646798 22022 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 11:12:14.646868 22022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260619 11:12:14.646956 22022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:14.647006 22022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.655601 22022 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:38955
I20260619 11:12:14.655638 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:38955 every 8 connection(s)
I20260619 11:12:14.656489 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:14.659256 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22022
I20260619 11:12:14.659698 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:35451
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=35297
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:14.671020 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.658231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:56243 (local address 127.7.90.126:37119)
0619 11:12:14.658363 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.658367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.659263 (+ 896us) server_negotiation.cc:407] Connection header received
0619 11:12:14.660553 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.660556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.660607 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.660679 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:14.662063 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.662917 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.664132 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.664598 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.664730 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.667113 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:14.667120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:14.667127 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:14.667165 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:14.668851 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.669322 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.669327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.669331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.669428 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.669789 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.669794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.669796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.670063 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:14.670176 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.670615 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.670766 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:12:14.671525 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.658592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:56243)
0619 11:12:14.659073 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.659134 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.660324 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.660886 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.660898 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.661364 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:14.661895 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.661909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.663046 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.663049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.663992 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.664000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.664740 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.664743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.664873 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.665497 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:14.665513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:14.666925 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:14.668963 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.668968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.668978 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.669222 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.669559 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.669567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.669569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.669691 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.670164 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:14.670169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:14.670371 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.670510 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.670731 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260619 11:12:14.672590 22152 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:14.672824 22152 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:14.673306 22152 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:14.674350 21831 ts_manager.cc:194] Registered new tserver with Master: 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
I20260619 11:12:14.674887 21831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:56243
W20260619 11:12:14.767119 22155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:14.767369 22155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:14.767442 22155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:14.771304 22155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:14.771401 22155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:14.771505 22155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:14.776562 22155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=35297
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:14.777561 22155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:14.778514 22155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:12:14.787803 22155 server_base.cc:1034] running on GCE node
W20260619 11:12:14.787827 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.787843 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:14.787949 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:14.788427 22155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:14.789000 22155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:14.790148 22155 hybrid_clock.cc:648] HybridClock initialized: now 1781867534790116 us; error 43 us; skew 500 ppm
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:14.792608 22155 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:14.793426 22155 webserver.cc:460] Webserver started at http://127.7.90.67:35297/ using document root <none> and password file <none>
I20260619 11:12:14.794037 22155 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:14.794139 22155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:14.797747 22155 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:12:14.799675 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.800833 22155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:14.800966 22155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "153cd8013dac4f9f832e587533c2de2b"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "6b8c6e4f0df1478ff0794bdf59080d43"
server_key_iv: "de2b864f0c70b594e21b885914c6d1c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:14.801497 22155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:14.824358 22155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:14.827373 22155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:14.827584 22155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:14.828158 22155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:14.828961 22155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:14.829033 22155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.829103 22155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:14.829174 22155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:14.838524 22155 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:35451
I20260619 11:12:14.838542 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:35451 every 8 connection(s)
I20260619 11:12:14.839479 22155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:14.846459 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22155
Jun 19 11:12:14 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867534, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:14.853205 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.841165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:48063 (local address 127.7.90.126:37119)
0619 11:12:14.841298 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.841302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.842204 (+ 902us) server_negotiation.cc:407] Connection header received
0619 11:12:14.843386 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.843388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.843434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.843507 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:14.844599 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.845101 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.846227 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.846427 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.846521 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.849449 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:14.849468 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:14.849474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:14.849504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:14.851000 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.851600 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.851605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.851609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.851698 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.852034 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.852041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.852046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.852279 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:14.852378 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.852962 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.853082 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:12:14.854307 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.841627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:48063)
0619 11:12:14.842058 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.842100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.843197 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.843669 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.843676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.844012 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:14.844480 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.844492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.845200 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.845202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.846130 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.846138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.846591 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.846596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.846781 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.847463 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:14.847490 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:14.849297 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:14.851141 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.851148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.851170 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.851478 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.851800 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:14.851802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:14.851803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:14.851936 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:14.852384 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:14.852388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:14.852679 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.852879 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.853216 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260619 11:12:14.855432 22284 heartbeater.cc:347] Connected to a master server at 127.7.90.126:37119
I20260619 11:12:14.855661 22284 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:14.856067 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.847206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.7.90.126:37119)
0619 11:12:14.847434 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.847438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.847537 (+ 99us) server_negotiation.cc:407] Connection header received
0619 11:12:14.847686 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.847688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.847721 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.847790 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:14.848644 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.849409 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.850754 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.850966 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.851046 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.852542 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:14.852545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:14.852548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:14.852571 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:14.854322 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.854838 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.854841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.854842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.854949 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:14.855277 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:14.855281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:14.855283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:14.855443 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:14.855651 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.855792 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.855901 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260619 11:12:14.856216 22284 heartbeater.cc:511] Master 127.7.90.126:37119 requested a full tablet report, sending...
I20260619 11:12:14.857338 21830 ts_manager.cc:194] Registered new tserver with Master: 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451)
I20260619 11:12:14.857954 21830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:48063
I20260619 11:12:14.869488 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:14.873471 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.870124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.7.90.126:37119)
0619 11:12:14.870251 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.870254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.870437 (+ 183us) server_negotiation.cc:407] Connection header received
0619 11:12:14.870570 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.870572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.870670 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.870755 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:14.870949 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.871442 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.872231 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.872448 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.872527 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.872783 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:14.873179 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:14.873267 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.873316 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.873364 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:12:14.874197 21830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:12:14.877460 21830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:12:14.891073 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.885096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:35451 (local address 127.0.0.1:58884)
0619 11:12:14.885751 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.885800 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.885962 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.887051 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.887055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.887075 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:14.887183 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.887190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888176 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.888182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.889643 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.889652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890525 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.890528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.890632 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.890666 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.890803 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.890921 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":3,"thread_start_us":129,"threads_started":1}
I20260619 11:12:14.891074 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.885271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.7.90.67:35451)
0619 11:12:14.885757 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.885769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.886106 (+ 337us) server_negotiation.cc:407] Connection header received
0619 11:12:14.886186 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.886192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.886486 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.886782 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:14.887311 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888044 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.889768 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890392 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.890518 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.890665 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.890832 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.890920 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":157,"threads_started":1}
I20260619 11:12:14.891384 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.885300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.0.0.1:52640)
0619 11:12:14.885912 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.885927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.886035 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.886887 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.886892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.886914 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:14.887066 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.887076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888684 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.888687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.889814 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.889823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890915 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.890919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.891069 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.891092 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.891156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.891218 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":94,"thread_start_us":93,"threads_started":1}
I20260619 11:12:14.891599 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.885929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.0.0.1:34804)
0619 11:12:14.886219 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.886231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.886310 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.887112 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.887114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.887126 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:14.887221 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.887230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888656 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.888660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.889820 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.889827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890915 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.890919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.891292 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.891312 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.891385 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.891460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":276,"thread_start_us":95,"threads_started":1}
I20260619 11:12:14.891999 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.886329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.7.90.65:36907)
0619 11:12:14.886681 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.886692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.886713 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:12:14.886786 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.886793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.886954 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.887147 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:14.887299 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888490 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.889940 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890770 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.891079 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.891229 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.891764 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.891862 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260619 11:12:14.892043 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.886012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.7.90.66:38955)
0619 11:12:14.886415 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.886426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.886446 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:12:14.886531 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.886536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.886716 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.887211 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:14.887346 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.888488 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.889969 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.890770 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.891639 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.891785 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.891826 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.891899 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":99,"threads_started":1}
I20260619 11:12:14.893427 22086 tablet_service.cc:1467] Processing CreateTablet for tablet ed2ac45d037a47dcb246a9b67648fafc (DEFAULT_TABLE table=test-table [id=c24a44de7b9a49c2af07cb7bef3f1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:14.894524 22086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2ac45d037a47dcb246a9b67648fafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:14.894411 22218 tablet_service.cc:1467] Processing CreateTablet for tablet ed2ac45d037a47dcb246a9b67648fafc (DEFAULT_TABLE table=test-table [id=c24a44de7b9a49c2af07cb7bef3f1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:14.894596 21952 tablet_service.cc:1467] Processing CreateTablet for tablet ed2ac45d037a47dcb246a9b67648fafc (DEFAULT_TABLE table=test-table [id=c24a44de7b9a49c2af07cb7bef3f1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:14.895429 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2ac45d037a47dcb246a9b67648fafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:14.895435 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2ac45d037a47dcb246a9b67648fafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:14.898566 22044 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Bootstrap starting.
I20260619 11:12:14.900765 22044 tablet_bootstrap.cc:654] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:14.901633 22044 log.cc:824] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:14.903080 22301 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Bootstrap starting.
I20260619 11:12:14.903494 22300 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Bootstrap starting.
I20260619 11:12:14.904039 22044 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: No bootstrap required, opened a new log
I20260619 11:12:14.904214 22044 ts_tablet_manager.cc:1397] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:12:14.906016 22300 tablet_bootstrap.cc:654] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:14.906822 22300 log.cc:824] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:14.907529 22301 tablet_bootstrap.cc:654] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:14.908180 22044 raft_consensus.cc:348] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.908403 22044 raft_consensus.cc:374] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:14.908475 22044 raft_consensus.cc:729] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732a1b42d98b4029820d19bc4f3cfbb1, State: Initialized, Role: FOLLOWER
I20260619 11:12:14.908681 22301 log.cc:824] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:14.908963 22044 consensus_queue.cc:260] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.909195 22300 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: No bootstrap required, opened a new log
I20260619 11:12:14.909387 22300 ts_tablet_manager.cc:1397] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:12:14.909862 22044 ts_tablet_manager.cc:1428] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:12:14.910226 22152 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
I20260619 11:12:14.910712 22301 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: No bootstrap required, opened a new log
I20260619 11:12:14.910868 22301 ts_tablet_manager.cc:1397] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 11:12:14.913174 22300 raft_consensus.cc:348] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.913463 22300 raft_consensus.cc:374] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:14.913594 22300 raft_consensus.cc:729] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198b8769b39b441bac62077a8f5d742d, State: Initialized, Role: FOLLOWER
I20260619 11:12:14.914228 22300 consensus_queue.cc:260] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.914629 22301 raft_consensus.cc:348] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.914885 22301 raft_consensus.cc:374] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:14.914898 22018 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
I20260619 11:12:14.914937 22301 raft_consensus.cc:729] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153cd8013dac4f9f832e587533c2de2b, State: Initialized, Role: FOLLOWER
I20260619 11:12:14.915247 22300 ts_tablet_manager.cc:1428] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:12:14.915318 22301 consensus_queue.cc:260] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.915894 22284 heartbeater.cc:503] Master 127.7.90.126:37119 was elected leader, sending a full tablet report...
I20260619 11:12:14.916219 22301 ts_tablet_manager.cc:1428] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 11:12:14.966730 22019 tablet.cc:2369] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:14.980742 22306 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:14.980912 22306 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.982051 22306 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
I20260619 11:12:14.986083 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.982217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.7.90.67:60279)
0619 11:12:14.982388 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.982404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.982489 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.982794 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.982798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.982821 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:14.982973 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.982978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.984007 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.984010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.984870 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.984879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.985637 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985657 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.985762 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.985791 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.985876 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.985950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 11:12:14.986088 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.982332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:60279 (local address 127.7.90.65:36907)
0619 11:12:14.982453 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.982458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.982471 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:12:14.982626 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.982632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.982691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.982750 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:14.983147 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.983896 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985002 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.985467 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985632 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.985696 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.985890 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.985969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:12:14.986133 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.982286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.7.90.67:54707)
0619 11:12:14.982717 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:14.982750 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:14.982839 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:14.983079 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:14.983084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:14.983107 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:14.983210 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.983215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.984127 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.984131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.984999 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:14.985009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.985681 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:14.985805 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.985853 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 11:12:14.985931 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:12:14.986001 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":132,"threads_started":1}
I20260619 11:12:14.986150 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:14.982413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:54707 (local address 127.7.90.66:38955)
0619 11:12:14.982612 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:14.982616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:14.982844 (+ 228us) server_negotiation.cc:407] Connection header received
0619 11:12:14.982918 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:14.982920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:14.982959 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:14.983015 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:14.983310 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.984025 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985105 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:14.985569 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:14.985670 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:14.985730 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:14.985946 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 11:12:14.986011 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 11:12:14.986697 22106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2ac45d037a47dcb246a9b67648fafc" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" is_pre_election: true
I20260619 11:12:14.986692 21972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2ac45d037a47dcb246a9b67648fafc" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198b8769b39b441bac62077a8f5d742d" is_pre_election: true
I20260619 11:12:14.986979 21972 raft_consensus.cc:2454] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 0.
I20260619 11:12:14.986979 22106 raft_consensus.cc:2454] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 0.
I20260619 11:12:14.987473 22172 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 153cd8013dac4f9f832e587533c2de2b, 198b8769b39b441bac62077a8f5d742d, 732a1b42d98b4029820d19bc4f3cfbb1; no voters:
I20260619 11:12:14.987751 22306 raft_consensus.cc:2781] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:14.987825 22306 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:14.987855 22306 raft_consensus.cc:3037] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.988840 22306 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.989171 22306 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 election: Requested vote from peers 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
I20260619 11:12:14.989543 22106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2ac45d037a47dcb246a9b67648fafc" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
I20260619 11:12:14.989542 21972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2ac45d037a47dcb246a9b67648fafc" candidate_uuid: "153cd8013dac4f9f832e587533c2de2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198b8769b39b441bac62077a8f5d742d"
I20260619 11:12:14.989660 21972 raft_consensus.cc:3037] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.989660 22106 raft_consensus.cc:3037] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:14.990876 22106 raft_consensus.cc:2454] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 1.
I20260619 11:12:14.990876 21972 raft_consensus.cc:2454] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153cd8013dac4f9f832e587533c2de2b in term 1.
I20260619 11:12:14.991124 22172 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1; no voters:
I20260619 11:12:14.991292 22306 raft_consensus.cc:2781] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:14.991567 22306 raft_consensus.cc:686] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 1 LEADER]: Becoming Leader. State: Replica: 153cd8013dac4f9f832e587533c2de2b, State: Running, Role: LEADER
I20260619 11:12:14.991874 22306 consensus_queue.cc:237] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:14.995054 21831 catalog_manager.cc:5495] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 153cd8013dac4f9f832e587533c2de2b (127.7.90.67). New cstate: current_term: 1 leader_uuid: "153cd8013dac4f9f832e587533c2de2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20260619 11:12:15.011675 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.008204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.7.90.67:35451)
0619 11:12:15.008319 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:15.008326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:15.008393 (+ 67us) server_negotiation.cc:407] Connection header received
0619 11:12:15.008578 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:15.008581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:15.008641 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:15.008745 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:15.008975 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:15.009459 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:15.010275 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:15.010498 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:15.010601 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:15.010879 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:15.011279 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:15.011408 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:15.011466 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 11:12:15.011519 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:12:15.017210 22106 raft_consensus.cc:1261] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Refusing update from remote peer 153cd8013dac4f9f832e587533c2de2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:15.017254 21972 raft_consensus.cc:1261] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Refusing update from remote peer 153cd8013dac4f9f832e587533c2de2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:15.018029 22308 consensus_queue.cc:1035] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:15.018168 22306 consensus_queue.cc:1035] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:15.025238 22310 mvcc.cc:205] Tried to move back new op lower bound from 7298529423423688704 to 7298529423329452032. Current Snapshot: MvccSnapshot[applied={T|T < 7298529423423688704}]
I20260619 11:12:15.026156 22311 mvcc.cc:205] Tried to move back new op lower bound from 7298529423423688704 to 7298529423329452032. Current Snapshot: MvccSnapshot[applied={T|T < 7298529423423688704}]
I20260619 11:12:15.028373 22312 mvcc.cc:205] Tried to move back new op lower bound from 7298529423423688704 to 7298529423329452032. Current Snapshot: MvccSnapshot[applied={T|T < 7298529423423688704}]
I20260619 11:12:15.034247 21831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:15.034384 21831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:15.036867 21831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Sending DeleteTablet for 3 replicas of tablet ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:15.037818 22218 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2ac45d037a47dcb246a9b67648fafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:15 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:58884
I20260619 11:12:15.037856 22086 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2ac45d037a47dcb246a9b67648fafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:15 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:52640
I20260619 11:12:15.038465 21952 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2ac45d037a47dcb246a9b67648fafc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:15 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:34804
I20260619 11:12:15.039072 22321 tablet_replica.cc:330] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: stopping tablet replica
I20260619 11:12:15.039232 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21869
I20260619 11:12:15.041724 22320 tablet_replica.cc:330] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: stopping tablet replica
I20260619 11:12:15.042074 22320 raft_consensus.cc:2229] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:15.042416 22320 raft_consensus.cc:2258] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:15.044092 22320 ts_tablet_manager.cc:1905] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:15.045754 22319 tablet_replica.cc:330] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: stopping tablet replica
I20260619 11:12:15.046108 22319 raft_consensus.cc:2229] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:12:15.046540 22319 raft_consensus.cc:2258] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:12:15.047168 21815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:36907 (error 108)
I20260619 11:12:15.047578 22320 ts_tablet_manager.cc:1918] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:15.047693 22320 log.cc:1196] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:15.047967 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22022
I20260619 11:12:15.048218 22319 ts_tablet_manager.cc:1905] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:15.050098 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.049768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:36907 (local address 127.0.0.1:34818)
0619 11:12:15.049906 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.049983 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:12:15.050887 22319 ts_tablet_manager.cc:1918] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 11:12:15.050279 21815 catalog_manager.cc:4584] TS 198b8769b39b441bac62077a8f5d742d (127.7.90.65:36907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed2ac45d037a47dcb246a9b67648fafc: Network error: Client connection negotiation failed: client connection to 127.7.90.65:36907: connect: Connection refused (error 111)
I20260619 11:12:15.051100 22319 log.cc:1196] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:15.051474 22319 ts_tablet_manager.cc:1939] T ed2ac45d037a47dcb246a9b67648fafc P 153cd8013dac4f9f832e587533c2de2b: Deleting consensus metadata
I20260619 11:12:15.052654 21815 catalog_manager.cc:4841] TS 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): tablet ed2ac45d037a47dcb246a9b67648fafc (table test-table [id=c24a44de7b9a49c2af07cb7bef3f1f28]) successfully deleted
I20260619 11:12:15.054944 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.054708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.0.0.1:52648)
0619 11:12:15.054837 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.054882 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:38955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:12:15.055172 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22155
I20260619 11:12:15.061043 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 21798
I20260619 11:12:15.067535 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--webserver_interface=127.7.90.126
--webserver_port=38689
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:15.175236 22323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:15.175535 22323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:15.175607 22323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:15.179486 22323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:15.179574 22323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:15.179611 22323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:15.179668 22323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:15.179720 22323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:15.185035 22323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:15.186163 22323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:15.187161 22323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:15.195152 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.195183 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.195195 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:15.195611 22323 server_base.cc:1034] running on GCE node
I20260619 11:12:15.196156 22323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:15.197180 22323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:15.198333 22323 hybrid_clock.cc:648] HybridClock initialized: now 1781867535198318 us; error 27 us; skew 500 ppm
Jun 19 11:12:15 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:15.200927 22323 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:15.201835 22323 webserver.cc:460] Webserver started at http://127.7.90.126:38689/ using document root <none> and password file <none>
I20260619 11:12:15.202333 22323 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:15.202394 22323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:15.206018 22323 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:15.207996 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.209170 22323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:15.209304 22323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "23837494044e4974a5b3fcd6852367ed"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "5313bc5fa509942971ee3d1afc25f0e0"
server_key_iv: "eab857c6bf61b33776a26d466797cc0b"
server_key_version: "encryptionkey@0"
I20260619 11:12:15.209892 22323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:15.233227 22323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:15.236094 22323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:15.236295 22323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:15.243279 22323 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:37119
I20260619 11:12:15.243588 22390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:37119 every 8 connection(s)
I20260619 11:12:15.244361 22323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:15.248462 22391 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:12:15.249981 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap starting.
I20260619 11:12:15.253830 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22323
I20260619 11:12:15.254225 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:36907
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=37325
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:15.266942 22391 log.cc:824] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:15.277976 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:15.278349 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: Bootstrap complete.
I20260619 11:12:15.282455 22391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:15.282850 22391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Initialized, Role: FOLLOWER
I20260619 11:12:15.283324 22391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:15.283465 22391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:15.283558 22391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:15.283675 22391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:12:15.285738 22391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:15.286140 22391 leader_election.cc:302] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23837494044e4974a5b3fcd6852367ed; no voters:
I20260619 11:12:15.286448 22391 leader_election.cc:288] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:12:15.286602 22395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:12:15.286931 22395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [term 3 LEADER]: Becoming Leader. State: Replica: 23837494044e4974a5b3fcd6852367ed, State: Running, Role: LEADER
I20260619 11:12:15.287303 22395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } }
I20260619 11:12:15.287640 22391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:15.288681 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:15.288801 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:15.289558 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23837494044e4974a5b3fcd6852367ed. Latest consensus state: current_term: 3 leader_uuid: "23837494044e4974a5b3fcd6852367ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23837494044e4974a5b3fcd6852367ed" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 37119 } } }
I20260619 11:12:15.289646 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:15.292061 22411 catalog_manager.cc:1259] Loaded cluster ID: 90b7bf96ff014d3db375db2f1058e30f
I20260619 11:12:15.292143 22411 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: loading cluster ID for follower catalog manager: success
W20260619 11:12:15.292924 22411 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 23837494044e4974a5b3fcd6852367ed: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 11:12:15.293012 22411 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 11:12:15.293077 22404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:15.295814 22404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:15.296316 22404 catalog_manager.cc:1259] Loaded cluster ID: 90b7bf96ff014d3db375db2f1058e30f
I20260619 11:12:15.296377 22404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 11:12:15.297014 22404 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa390b3e9ad google::LogMessage::Fail() at ??:0
@ 0x7fa390b4188a google::LogMessage::SendToLog() at ??:0
@ 0x7fa390b3e4de google::LogMessage::Flush() at ??:0
@ 0x7fa390b3fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa39b47dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa39b47ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa39b47b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa39b4bf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa392d903d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa392d90c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa392d923f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa392d81512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3943c56db start_thread at ??:0
@ 0x7fa38f63c61f clone at ??:0
W20260619 11:12:15.392283 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:15.392520 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:15.392561 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:15.396634 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:15.396682 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:15.396764 22393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:15.401811 22393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:36907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=37325
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:15.402740 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:15.403683 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:15.412844 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.412863 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.412827 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:15.413467 22393 server_base.cc:1034] running on GCE node
I20260619 11:12:15.413937 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:15.414561 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:15.415740 22393 hybrid_clock.cc:648] HybridClock initialized: now 1781867535415711 us; error 46 us; skew 500 ppm
Jun 19 11:12:15 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:15.418710 22393 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:15.419889 22393 webserver.cc:460] Webserver started at http://127.7.90.65:37325/ using document root <none> and password file <none>
I20260619 11:12:15.420557 22393 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:15.420643 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:15.425707 22393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:12:15.428305 22442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.429919 22393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:12:15.430059 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "198b8769b39b441bac62077a8f5d742d"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "e75177bd5905b0c5e847f617b67b0f55"
server_key_iv: "139fcec1aeb24bfbd24ff45b3f3279b0"
server_key_version: "encryptionkey@0"
I20260619 11:12:15.430774 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:15.461266 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:15.465157 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:15.465375 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:15.466264 22393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:15.467712 22449 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:12:15.471935 22393 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:12:15.472021 22393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 11:12:15.472088 22393 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:12:15.474273 22393 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:12:15.474342 22393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:15.474720 22449 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Bootstrap starting.
I20260619 11:12:15.485531 22393 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:36907
I20260619 11:12:15.485577 22556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:36907 every 8 connection(s)
I20260619 11:12:15.486740 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:15.491014 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22393
I20260619 11:12:15.491472 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:38955
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=39243
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:15.494976 22449 log.cc:824] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:15.502207 22449 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:15.502645 22449 tablet_bootstrap.cc:492] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Bootstrap complete.
I20260619 11:12:15.503769 22449 ts_tablet_manager.cc:1397] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20260619 11:12:15.506325 22449 raft_consensus.cc:348] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:15.506747 22449 raft_consensus.cc:729] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198b8769b39b441bac62077a8f5d742d, State: Initialized, Role: FOLLOWER
I20260619 11:12:15.507247 22449 consensus_queue.cc:260] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:15.508198 22449 ts_tablet_manager.cc:1428] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260619 11:12:15.538906 22559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:12:15.489095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:42399)
0619 11:12:15.489896 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.489968 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:15.491330 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:15.538167 (+ 46837us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":143,"threads_started":1}
W20260619 11:12:15.539499 22557 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37119 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:37119: Client connection negotiation failed: client connection to 127.7.90.126:37119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:12:15.540217 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.539957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:38823)
0619 11:12:15.540090 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.540165 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 11:12:15.540932 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.540733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:46013)
0619 11:12:15.540838 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.540888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 11:12:15.541122 22557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:12:15.614670 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:15.614967 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:15.615063 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:15.619195 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:15.619277 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:15.619372 22560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:15.624496 22560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=39243
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:15.625499 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:15.626493 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:15.636322 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.636312 22570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.636314 22568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:15.636742 22560 server_base.cc:1034] running on GCE node
I20260619 11:12:15.637125 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:15.637805 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:15.638994 22560 hybrid_clock.cc:648] HybridClock initialized: now 1781867535638968 us; error 45 us; skew 500 ppm
Jun 19 11:12:15 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:15.641454 22560 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:15.642382 22560 webserver.cc:460] Webserver started at http://127.7.90.66:39243/ using document root <none> and password file <none>
I20260619 11:12:15.642928 22560 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:15.643016 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:15.646675 22560 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:12:15.648577 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.649801 22560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:15.649938 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "732a1b42d98b4029820d19bc4f3cfbb1"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "340b9e8d27bc14e76442dc5b34c4ffae"
server_key_iv: "f5b2ebc0d8fe1d930649e9dcef9790c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:15.650452 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:15.664871 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:15.667615 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:15.667809 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:15.668375 22560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:15.669445 22584 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 11:12:15.673024 22584 ts_tablet_manager.cc:1852] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 11:12:15.673400 22584 ts_tablet_manager.cc:1905] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:15.676942 22584 ts_tablet_manager.cc:1918] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:15.677132 22584 ts_tablet_manager.cc:1939] T ed2ac45d037a47dcb246a9b67648fafc P 732a1b42d98b4029820d19bc4f3cfbb1: Deleting consensus metadata
I20260619 11:12:15.677440 22560 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 11:12:15.677515 22560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260619 11:12:15.677620 22560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:15.677664 22560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.686714 22560 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:38955
I20260619 11:12:15.686751 22691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:38955 every 8 connection(s)
I20260619 11:12:15.687541 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:15.689150 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22560
I20260619 11:12:15.689572 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:35451
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:37119
--webserver_port=35297
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:15.691244 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.689615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:49715)
0619 11:12:15.690202 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.690339 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":123,"threads_started":1}
W20260619 11:12:15.691664 22692 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37119 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:37119: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
I20260619 11:12:15.692270 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.692078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:57909)
0619 11:12:15.692181 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.692222 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:12:15.692935 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.692758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:57003)
0619 11:12:15.692852 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.692890 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:12:15.693128 22692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:12:15.795753 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:15.796098 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:15.796173 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:15.800043 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:15.800091 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:15.800146 22695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:15.805459 22695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=35297
--tserver_master_addrs=127.7.90.126:37119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:15.806526 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:15.807510 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:15.816846 22700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:15.816874 22703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:15.817070 22695 server_base.cc:1034] running on GCE node
W20260619 11:12:15.816879 22701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:15.817528 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:15.818149 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:15.819329 22695 hybrid_clock.cc:648] HybridClock initialized: now 1781867535819315 us; error 25 us; skew 500 ppm
Jun 19 11:12:15 dist-test-slave-f1t5 krb5kdc[21251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:15.821897 22695 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:15.822692 22695 webserver.cc:460] Webserver started at http://127.7.90.67:35297/ using document root <none> and password file <none>
I20260619 11:12:15.823240 22695 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:15.823331 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:15.826988 22695 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:15.828979 22710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.830209 22695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:15.830341 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "153cd8013dac4f9f832e587533c2de2b"
format_stamp: "Formatted at 2026-06-19 11:12:13 on dist-test-slave-f1t5"
server_key: "6b8c6e4f0df1478ff0794bdf59080d43"
server_key_iv: "de2b864f0c70b594e21b885914c6d1c9"
server_key_version: "encryptionkey@0"
I20260619 11:12:15.830861 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:15.875828 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:15.878988 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:15.879225 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:15.879842 22695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:15.880685 22695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:15.880767 22695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.880841 22695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:15.880877 22695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:15.890025 22695 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:35451
I20260619 11:12:15.890044 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:35451 every 8 connection(s)
I20260619 11:12:15.891077 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:15.894886 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.893200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:37111)
0619 11:12:15.893710 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.894203 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
W20260619 11:12:15.895373 22824 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:37119 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:37119: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
I20260619 11:12:15.895905 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.895705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:38521)
0619 11:12:15.895810 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.895855 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 11:12:15.896505 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:15.896328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:53347)
0619 11:12:15.896419 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:15.896461 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:12:15.896750 22824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:12:15.896893 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22695
I20260619 11:12:16.542155 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:16.541705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:44089)
0619 11:12:16.541971 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:16.542044 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260619 11:12:16.694103 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:16.693720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:41815)
0619 11:12:16.693968 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:16.694017 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260619 11:12:16.897915 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:16.897391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:35141)
0619 11:12:16.897726 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:16.897792 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":146,"threads_started":1}
I20260619 11:12:17.198683 22831 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:17.198938 22831 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:17.200153 22831 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
Jun 19 11:12:17 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.67@KRBTEST.COM
Jun 19 11:12:17 dist-test-slave-f1t5 krb5kdc[21251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867535, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.66@KRBTEST.COM
I20260619 11:12:17.217980 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.200503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:44267 (local address 127.7.90.67:35451)
0619 11:12:17.201142 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:17.201164 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:17.201199 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:12:17.202013 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:17.202034 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:17.202535 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:17.202895 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:17.204875 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.206533 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.208630 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.209410 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.209629 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:17.213003 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:17.213014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:17.213036 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:17.213074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:17.214903 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:17.215558 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:17.215564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:17.215570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:17.215725 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:17.216159 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:17.216162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:17.216164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:17.216697 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:17.216798 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:17.217439 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 11:12:17.217775 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":95,"threads_started":1}
I20260619 11:12:17.217989 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.200657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:38093 (local address 127.7.90.66:38955)
0619 11:12:17.200991 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:17.201009 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:17.201059 (+ 50us) server_negotiation.cc:407] Connection header received
0619 11:12:17.202072 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:17.202080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:17.202539 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:17.202895 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:17.204860 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.206533 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.208630 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.209410 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.209629 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:17.213291 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:17.213298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:17.213318 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:17.213364 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:17.215421 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:17.215917 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:17.215923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:17.215928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:17.216057 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:17.216391 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:17.216394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:17.216395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:17.216869 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:17.216991 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:17.217442 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 11:12:17.217744 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":133,"threads_started":1}
I20260619 11:12:17.217981 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.200341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:35451 (local address 127.7.90.65:44267)
0619 11:12:17.200916 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:17.200929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:17.201005 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:17.203021 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:17.203028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:17.203671 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:17.204614 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:17.204638 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.206739 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.206743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:17.208462 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:17.208475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.209583 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.209587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:17.209908 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:17.210884 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:17.210927 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:17.212700 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:17.215039 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:17.215046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:17.215060 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:17.215418 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:17.215836 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:17.215839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:17.215841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:17.216058 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:17.216847 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:17.216854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:17.217131 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 11:12:17.217331 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:12:17.217755 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":577,"spinlock_wait_cycles":33920,"thread_start_us":64,"threads_started":1}
I20260619 11:12:17.218020 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.200341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:38955 (local address 127.7.90.65:38093)
0619 11:12:17.200790 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:17.200806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:17.200924 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:17.203021 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:17.203028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:17.203671 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:17.204614 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:17.204638 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.206739 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.206743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:17.208462 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:17.208475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:17.209583 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:17.209587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:17.209908 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:17.210884 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:17.210914 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:17.213082 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:17.215534 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:17.215537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:17.215539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:17.215794 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:17.216167 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:17.216171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:17.216173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:17.216272 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:17.216985 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:17.216988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:17.217129 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 11:12:17.217330 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 11:12:17.217841 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":593,"spinlock_wait_cycles":118016,"thread_start_us":72,"threads_started":1}
W20260619 11:12:17.220710 22444 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
W20260619 11:12:17.221019 22443 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:17.221123 22443 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198b8769b39b441bac62077a8f5d742d; no voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1
I20260619 11:12:17.221467 22831 raft_consensus.cc:2726] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:12:17.543329 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.542994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:40523)
0619 11:12:17.543162 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:17.543224 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 11:12:17.695375 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.694894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:56373)
0619 11:12:17.695178 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:17.695257 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260619 11:12:17.898994 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:17.898617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:37657)
0619 11:12:17.898859 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:17.898911 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260619 11:12:18.544500 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:18.544049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:47971)
0619 11:12:18.544306 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:18.544391 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260619 11:12:18.696791 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:18.696327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:49095)
0619 11:12:18.696597 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:18.696657 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260619 11:12:18.900067 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:18.899662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:48375)
0619 11:12:18.899917 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:18.899970 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260619 11:12:19.133302 22841 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:19.133471 22841 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:19.133989 22841 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
W20260619 11:12:19.134820 22444 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
W20260619 11:12:19.134923 22443 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:19.134985 22443 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198b8769b39b441bac62077a8f5d742d; no voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1
I20260619 11:12:19.135128 22841 raft_consensus.cc:2726] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:12:19.545797 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:19.545204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:43809)
0619 11:12:19.545630 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:19.545685 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":90,"threads_started":1}
I20260619 11:12:19.698029 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:19.697638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:34413)
0619 11:12:19.697875 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:19.697926 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260619 11:12:19.901109 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:19.900740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:52035)
0619 11:12:19.900982 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:19.901030 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
I20260619 11:12:20.547017 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:20.546610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:52321)
0619 11:12:20.546873 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:20.546923 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260619 11:12:20.699188 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:20.698769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:35747)
0619 11:12:20.699049 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:20.699102 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260619 11:12:20.902246 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:20.901835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:44169)
0619 11:12:20.902097 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:20.902154 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260619 11:12:21.548273 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:21.547848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:48623)
0619 11:12:21.548123 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:21.548181 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260619 11:12:21.700493 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:21.700081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:35755)
0619 11:12:21.700339 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:21.700399 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260619 11:12:21.903373 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:21.902976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:40223)
0619 11:12:21.903210 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:21.903267 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260619 11:12:22.315557 22851 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:22.315706 22851 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:22.316210 22851 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
W20260619 11:12:22.317062 22443 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
W20260619 11:12:22.317234 22444 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:22.317306 22444 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198b8769b39b441bac62077a8f5d742d; no voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1
I20260619 11:12:22.317477 22851 raft_consensus.cc:2726] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:12:22.549508 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:22.549122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:46845)
0619 11:12:22.549361 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:22.549415 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260619 11:12:22.701794 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:22.701322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:42509)
0619 11:12:22.701650 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:22.701706 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":73,"threads_started":1}
I20260619 11:12:22.904435 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:22.904051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:48197)
0619 11:12:22.904305 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:22.904352 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":65,"threads_started":1}
I20260619 11:12:23.550833 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:23.550373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:45143)
0619 11:12:23.550670 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:23.550734 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260619 11:12:23.702819 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:23.702454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:41615)
0619 11:12:23.702682 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:23.702731 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260619 11:12:23.905637 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:23.905170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:54717)
0619 11:12:23.905416 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:23.905482 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260619 11:12:24.552098 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:24.551655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:40991)
0619 11:12:24.551951 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:24.552004 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260619 11:12:24.703964 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:24.703566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:51885)
0619 11:12:24.703818 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:24.703874 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260619 11:12:24.906699 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:24.906303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:48509)
0619 11:12:24.906558 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:24.906611 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260619 11:12:25.553503 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:25.553093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:50561)
0619 11:12:25.553340 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:25.553395 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260619 11:12:25.705298 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:25.704889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:38435)
0619 11:12:25.705154 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:25.705206 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260619 11:12:25.744717 22863 raft_consensus.cc:482] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:25.744849 22863 raft_consensus.cc:504] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153cd8013dac4f9f832e587533c2de2b" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 35451 } } peers { permanent_uuid: "198b8769b39b441bac62077a8f5d742d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 36907 } } peers { permanent_uuid: "732a1b42d98b4029820d19bc4f3cfbb1" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 38955 } }
I20260619 11:12:25.745227 22863 leader_election.cc:288] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451), 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955)
W20260619 11:12:25.745985 22443 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153cd8013dac4f9f832e587533c2de2b (127.7.90.67:35451): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
W20260619 11:12:25.746090 22444 leader_election.cc:341] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732a1b42d98b4029820d19bc4f3cfbb1 (127.7.90.66:38955): Not found: Tablet not found: ed2ac45d037a47dcb246a9b67648fafc
I20260619 11:12:25.746189 22444 leader_election.cc:302] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198b8769b39b441bac62077a8f5d742d; no voters: 153cd8013dac4f9f832e587533c2de2b, 732a1b42d98b4029820d19bc4f3cfbb1
I20260619 11:12:25.746349 22863 raft_consensus.cc:2726] T ed2ac45d037a47dcb246a9b67648fafc P 198b8769b39b441bac62077a8f5d742d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:12:25.907814 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:25.907419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:58037)
0619 11:12:25.907666 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:25.907722 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260619 11:12:26.554708 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:26.554278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:59405)
0619 11:12:26.554544 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:26.554611 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260619 11:12:26.706357 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:26.705968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:34553)
0619 11:12:26.706210 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:26.706272 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260619 11:12:26.908932 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:26.908563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:57357)
0619 11:12:26.908788 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:26.908848 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260619 11:12:27.555804 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:27.555419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:45257)
0619 11:12:27.555656 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:27.555710 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260619 11:12:27.707471 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:27.706994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:40401)
0619 11:12:27.707319 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:27.707375 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":69,"threads_started":1}
I20260619 11:12:27.910063 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:27.909691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:33243)
0619 11:12:27.909914 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:27.909966 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260619 11:12:28.556900 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:28.556507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:37263)
0619 11:12:28.556747 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:28.556799 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260619 11:12:28.708564 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:28.708174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:34167)
0619 11:12:28.708402 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:28.708456 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260619 11:12:28.911249 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:28.910838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:33131)
0619 11:12:28.911098 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:28.911159 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260619 11:12:29.558030 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:29.557627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:48937)
0619 11:12:29.557880 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:29.557935 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260619 11:12:29.709853 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:29.709419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:55053)
0619 11:12:29.709709 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:29.709762 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
I20260619 11:12:29.912568 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:29.912116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.67:39869)
0619 11:12:29.912432 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:29.912484 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260619 11:12:30.559271 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:30.558878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.65:43045)
0619 11:12:30.559130 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:30.559179 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":107,"threads_started":1}
I20260619 11:12:30.711055 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:30.710652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:37119 (local address 127.7.90.66:56697)
0619 11:12:30.710902 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:30.710951 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260619 11:12:30.901126 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22393
I20260619 11:12:30.906833 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22560
I20260619 11:12:30.912531 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22695
2026-06-19T11:12:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19149 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:12:30 dist-test-slave-f1t5 krb5kdc[22881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:12:30 dist-test-slave-f1t5 krb5kdc[22881](info): set up 2 sockets
Jun 19 11:12:30 dist-test-slave-f1t5 krb5kdc[22881](info): commencing operation
krb5kdc: starting...
W20260619 11:12:31.953907 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:12:32.067085 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:45413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:32.172880 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:32.173167 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:32.173238 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:32.177179 22894 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:12:32.177256 22894 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:12:32.177281 22894 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:12:32.177304 22894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:32.177341 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:32.177380 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:32.177416 22894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:32.177454 22894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:32.182539 22894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:45413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:32.183532 22894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:32.184521 22894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:32.192842 22900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.192924 22902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.192883 22899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.192907 22894 server_base.cc:1034] running on GCE node
I20260619 11:12:32.193730 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:32.194703 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:32.195858 22894 hybrid_clock.cc:648] HybridClock initialized: now 1781867552195843 us; error 26 us; skew 500 ppm
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:32.198458 22894 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:32.199203 22894 webserver.cc:460] Webserver started at http://127.7.90.126:33945/ using document root <none> and password file <none>
I20260619 11:12:32.199700 22894 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:32.199783 22894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:32.199991 22894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:32.201951 22894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "2a5c29ed4d5f442999fc2b5077e1fa40"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "23397000aa53455434c38159c6ddf905"
server_key_iv: "1636c0b536bb2563bc818d96e3a73b39"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.202667 22894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "2a5c29ed4d5f442999fc2b5077e1fa40"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "23397000aa53455434c38159c6ddf905"
server_key_iv: "1636c0b536bb2563bc818d96e3a73b39"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.206248 22894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:12:32.208456 22909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.209647 22894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:32.209767 22894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2a5c29ed4d5f442999fc2b5077e1fa40"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "23397000aa53455434c38159c6ddf905"
server_key_iv: "1636c0b536bb2563bc818d96e3a73b39"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.209865 22894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:32.220705 22894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:32.223762 22894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:32.223961 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:32.230796 22894 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:45413
I20260619 11:12:32.231112 22961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:45413 every 8 connection(s)
I20260619 11:12:32.231710 22894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:32.232178 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22894
I20260619 11:12:32.232309 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:12:32.232558 7529 external_mini_cluster.cc:1428] Setting key 09135a2a80796f7e1ee9ab73ecf7d32f
I20260619 11:12:32.234714 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:32.239840 22962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap starting.
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:32.242131 22962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:32.242935 22962 log.cc:824] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:32.244964 22962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: No bootstrap required, opened a new log
I20260619 11:12:32.246197 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.233886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.7.90.126:45413)
0619 11:12:32.234295 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.234309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.234341 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:12:32.235132 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.235148 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.235463 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.235817 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:32.236600 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.237632 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.238496 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.238950 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.239110 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.241599 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:32.241605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:32.241618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:32.241642 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:32.243459 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.243968 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.243973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.243977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.244074 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.244359 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.244361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.244363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.244736 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:32.244873 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.245214 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.245506 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260619 11:12:32.248725 22962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:32.248914 22962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:32.248982 22962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Initialized, Role: FOLLOWER
I20260619 11:12:32.249365 22962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:32.249505 22962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:32.249589 22962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:32.249698 22962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:32.250463 22962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:32.250779 22962 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5c29ed4d5f442999fc2b5077e1fa40; no voters:
I20260619 11:12:32.251044 22962 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:12:32.251148 22967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:32.251382 22967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Running, Role: LEADER
I20260619 11:12:32.251729 22967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:32.252182 22962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:32.253285 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5c29ed4d5f442999fc2b5077e1fa40. Latest consensus state: current_term: 1 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:12:32.253405 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:32.253794 22968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:12:32.253943 22968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:32.254493 22976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:32.257073 22976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:32.261684 22976 catalog_manager.cc:1347] Generated new cluster ID: 797b6955bfb54d6da95b69c06d2b1395
I20260619 11:12:32.261745 22976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:32.288838 22976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:12:32.289945 22976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:32.303928 22976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Generated new TSK 0
I20260619 11:12:32.304702 22976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:12:32.334609 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:32.442963 22988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:32.443356 22988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:32.443429 22988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:32.447333 22988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:32.447384 22988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:32.447489 22988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:32.452821 22988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:32.453956 22988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:32.454981 22988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:12:32.464346 22988 server_base.cc:1034] running on GCE node
W20260619 11:12:32.464303 22993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.464244 22994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.464306 22996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.464892 22988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:32.465503 22988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:32.466728 22988 hybrid_clock.cc:648] HybridClock initialized: now 1781867552466654 us; error 87 us; skew 500 ppm
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:32.469254 22988 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:32.470069 22988 webserver.cc:460] Webserver started at http://127.7.90.65:39527/ using document root <none> and password file <none>
I20260619 11:12:32.470558 22988 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:32.470611 22988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:32.470762 22988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:32.472743 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "a0eb28bca444418fac377b397394d86c"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "2128b76be74dcab6ce9622451a9b7ab3"
server_key_iv: "cf3980ebaac52626423d4c21d844ebd1"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.473238 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "a0eb28bca444418fac377b397394d86c"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "2128b76be74dcab6ce9622451a9b7ab3"
server_key_iv: "cf3980ebaac52626423d4c21d844ebd1"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.476687 22988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:12:32.478963 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.480116 22988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:32.480202 22988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "a0eb28bca444418fac377b397394d86c"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "2128b76be74dcab6ce9622451a9b7ab3"
server_key_iv: "cf3980ebaac52626423d4c21d844ebd1"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.480279 22988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:32.500156 22988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:32.503067 22988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:32.503207 22988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:32.503674 22988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:32.504386 22988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:32.504436 22988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.504473 22988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:32.504488 22988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.513151 22988 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:37447
I20260619 11:12:32.513155 23116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:37447 every 8 connection(s)
I20260619 11:12:32.514081 22988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:32.520396 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 22988
I20260619 11:12:32.520545 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:12:32.520859 7529 external_mini_cluster.cc:1428] Setting key 0b029d41cd67e09ce4bc086f30b15099
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:32.527334 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.515794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:38611 (local address 127.7.90.126:45413)
0619 11:12:32.515925 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.515936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.516856 (+ 920us) server_negotiation.cc:407] Connection header received
0619 11:12:32.517730 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.517734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.517793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.517871 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:32.518952 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.519487 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.520857 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.521143 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.521258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.523799 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:32.523803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:32.523805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:32.523832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:32.525179 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.525811 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.525814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.525815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.525890 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.526270 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.526272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.526273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.526431 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:32.526513 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.527119 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.527221 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:12:32.528184 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.516100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:38611)
0619 11:12:32.516711 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.516750 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.517533 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.518035 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.518042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.518371 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:32.518810 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.518822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.519620 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.519626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.520740 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.520752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.521295 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.521299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.521465 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.522100 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:32.522116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:32.523634 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:32.525295 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.525304 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.525320 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.525699 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.525983 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.525989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.525994 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.526173 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.526534 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:32.526544 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:32.526848 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.527071 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.527404 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":106,"threads_started":1}
I20260619 11:12:32.529335 23117 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:32.529601 23117 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:32.530143 23117 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:12:32.531666 22926 ts_manager.cc:194] Registered new tserver with Master: a0eb28bca444418fac377b397394d86c (127.7.90.65:37447)
I20260619 11:12:32.532742 22926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:38611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:12:32.547817 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:32.653084 23122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:32.653322 23122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:32.653362 23122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:32.657280 23122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:32.657332 23122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:32.657384 23122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:32.662269 23122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:32.663115 23122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:32.663977 23122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:32.672818 23128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.672852 23127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.672935 23122 server_base.cc:1034] running on GCE node
W20260619 11:12:32.673115 23130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.673475 23122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:32.674129 23122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:32.675292 23122 hybrid_clock.cc:648] HybridClock initialized: now 1781867552675276 us; error 26 us; skew 500 ppm
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:32.677932 23122 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:32.678745 23122 webserver.cc:460] Webserver started at http://127.7.90.66:46517/ using document root <none> and password file <none>
I20260619 11:12:32.679260 23122 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:32.679342 23122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:32.679550 23122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:32.681382 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "c40e8a487390d80c04ece06ec047593b"
server_key_iv: "b0c5d12e20f10f6e8ec5911760a8c78f"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.681984 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "c40e8a487390d80c04ece06ec047593b"
server_key_iv: "b0c5d12e20f10f6e8ec5911760a8c78f"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.685328 23122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:12:32.687595 23137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.688731 23122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:32.688855 23122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "c40e8a487390d80c04ece06ec047593b"
server_key_iv: "b0c5d12e20f10f6e8ec5911760a8c78f"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.688958 23122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:32.699594 23122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:32.702493 23122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:32.702677 23122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:32.703178 23122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:32.703886 23122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:32.703954 23122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.704026 23122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:32.704067 23122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.712576 23122 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:37767
I20260619 11:12:32.712620 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:37767 every 8 connection(s)
I20260619 11:12:32.713452 23122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:32.723079 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23122
I20260619 11:12:32.723214 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:12:32.723485 7529 external_mini_cluster.cc:1428] Setting key ee24a06259baf2262ec6ca44ea6d7311
I20260619 11:12:32.726416 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.715096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:47837 (local address 127.7.90.126:45413)
0619 11:12:32.715217 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.715221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.716376 (+ 1155us) server_negotiation.cc:407] Connection header received
0619 11:12:32.717216 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.717220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.717275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.717366 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:32.718452 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.719002 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.720113 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.720341 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.720443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.722756 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:32.722759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:32.722761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:32.722783 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:32.724295 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.724821 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.724825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.724827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.724935 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.725308 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.725310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.725311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.725475 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:32.725628 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.726215 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.726307 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:12:32.727275 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.715424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:47837)
0619 11:12:32.716247 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.716285 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.717022 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.717567 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.717576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.717893 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:32.718301 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.718313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.719119 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.719123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.719971 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.719979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.720494 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.720497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.720647 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.721207 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:32.721221 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:32.722585 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:32.724397 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.724401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.724410 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.724713 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.725042 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.725045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.725047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.725212 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.725648 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:32.725657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:32.725978 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.726170 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.726492 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":526,"threads_started":1}
I20260619 11:12:32.728433 23251 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:32.728655 23251 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:32.729235 23251 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
I20260619 11:12:32.730356 22926 ts_manager.cc:194] Registered new tserver with Master: 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
I20260619 11:12:32.730891 22926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:47837
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:12:32.749504 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:32.858258 23256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:32.858490 23256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:32.858529 23256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:32.862509 23256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:32.862558 23256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:32.862610 23256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:32.867841 23256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:32.868739 23256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:32.869753 23256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:32.878767 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:32.878782 23261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.878844 23256 server_base.cc:1034] running on GCE node
W20260619 11:12:32.878794 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:32.879366 23256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:32.879961 23256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:32.881145 23256 hybrid_clock.cc:648] HybridClock initialized: now 1781867552881111 us; error 47 us; skew 500 ppm
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:32.883908 23256 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:32.884730 23256 webserver.cc:460] Webserver started at http://127.7.90.67:39155/ using document root <none> and password file <none>
I20260619 11:12:32.885259 23256 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:32.885344 23256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:32.885614 23256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:12:32.887509 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "47bcee575eb24eb4952f75ec57d25592"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "e3baf15aadd18f29348ddfcf113ec39e"
server_key_iv: "19a54e742bf36bf58567f9ff35923c4c"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.888055 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "47bcee575eb24eb4952f75ec57d25592"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "e3baf15aadd18f29348ddfcf113ec39e"
server_key_iv: "19a54e742bf36bf58567f9ff35923c4c"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.891476 23256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:12:32.893731 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.894833 23256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:12:32.894960 23256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "47bcee575eb24eb4952f75ec57d25592"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "e3baf15aadd18f29348ddfcf113ec39e"
server_key_iv: "19a54e742bf36bf58567f9ff35923c4c"
server_key_version: "encryptionkey@0"
I20260619 11:12:32.895068 23256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:32.925060 23256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:32.928228 23256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:32.928431 23256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:32.928977 23256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:32.929811 23256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:32.929876 23256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.929942 23256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:32.929986 23256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:32.938503 23256 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:36949
I20260619 11:12:32.938549 23384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:36949 every 8 connection(s)
I20260619 11:12:32.939327 23256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:32.944725 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23256
I20260619 11:12:32.944851 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:12:32.945106 7529 external_mini_cluster.cc:1428] Setting key c990db7087fba5031ea7f5e53b14e9b4
Jun 19 11:12:32 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867552, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:32.952399 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.940986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:38755 (local address 127.7.90.126:45413)
0619 11:12:32.941136 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.941142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.942067 (+ 925us) server_negotiation.cc:407] Connection header received
0619 11:12:32.942930 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.942934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.942989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.943102 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:32.944245 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.944752 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.946231 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.946528 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.946642 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.948992 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:32.949003 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:32.949006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:32.949038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:32.950447 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.950936 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.950941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.950943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.951027 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.951353 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.951356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.951357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.951528 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:32.951627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.952113 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.952225 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 11:12:32.953190 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.941311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:38755)
0619 11:12:32.941929 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.941968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.942715 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.943266 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.943272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.943633 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:32.944109 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.944121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.944870 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.944875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.946116 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.946125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.946674 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.946678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.946861 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.947460 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:32.947475 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:32.948843 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:32.950580 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.950584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.950596 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.950832 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.951122 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:32.951124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:32.951126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:32.951255 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:32.951657 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:32.951665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:32.951894 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.952072 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.952368 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":124,"threads_started":1}
I20260619 11:12:32.954244 23385 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:32.954464 23385 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:32.954905 23385 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
I20260619 11:12:32.955825 22926 ts_manager.cc:194] Registered new tserver with Master: 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949)
I20260619 11:12:32.956369 22926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:38755
I20260619 11:12:32.958696 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:32.967973 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.960535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.7.90.126:45413)
0619 11:12:32.960668 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.960672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.960873 (+ 201us) server_negotiation.cc:407] Connection header received
0619 11:12:32.961023 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.961026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.961069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.961150 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:32.961914 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.962406 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.963201 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.963428 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.963563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.964938 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:32.964942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:32.964944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:32.964973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:32.966391 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.966804 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.966809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.966811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.966888 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:32.967157 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:32.967162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:32.967163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:32.967359 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:32.967495 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.967728 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.967825 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:12:32.970629 22926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:12:32.973730 22926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:12:32.987399 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.982170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:37767 (local address 127.0.0.1:44338)
0619 11:12:32.982653 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.982666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.982765 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.983378 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.983383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.983404 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:32.983543 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.983551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.984595 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.984599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.985983 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.985994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.986878 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.986881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.986986 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987035 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.987139 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.987263 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260619 11:12:32.987408 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.981499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:36949 (local address 127.0.0.1:44264)
0619 11:12:32.981992 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.982038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.982157 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.983161 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.983166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.983189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:32.983295 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.983299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.984871 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.984875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.986062 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.986073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.987099 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.987105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.987202 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987220 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.987273 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.987324 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":58,"thread_start_us":62,"threads_started":1}
I20260619 11:12:32.987816 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.981650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.7.90.67:36949)
0619 11:12:32.982615 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.982630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.982656 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:12:32.982742 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.982747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.982948 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.983461 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:32.983603 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.984738 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.986174 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.986949 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.987098 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987247 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.987297 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.987382 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":113,"threads_started":1}
I20260619 11:12:32.987829 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.981665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.0.0.1:52926)
0619 11:12:32.982262 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:32.982279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:32.982367 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:32.983602 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:32.983607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:32.983625 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:32.983718 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.983723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.985084 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.985089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.986254 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:32.986264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.987441 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.987445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:32.987560 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987591 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 11:12:32.987661 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:12:32.987717 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":28,"thread_start_us":103,"threads_started":1}
I20260619 11:12:32.987926 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.982208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.7.90.65:37447)
0619 11:12:32.982988 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.983018 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.983036 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:12:32.983098 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.983102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.983238 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.983393 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:32.983840 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.984967 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.986377 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.987157 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.987261 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987371 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.987709 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.987804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":75,"threads_started":1}
I20260619 11:12:32.988035 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:32.982507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.7.90.66:37767)
0619 11:12:32.982929 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:32.982940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:32.982960 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:12:32.983050 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:32.983055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:32.983236 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:32.983393 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:32.983626 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.984423 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.986109 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:32.986761 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:32.986881 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:32.987018 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:32.987813 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 11:12:32.987891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":91,"threads_started":1}
I20260619 11:12:32.990613 23319 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d6869f01340579b37c459a66d3257 (DEFAULT_TABLE table=test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:32.990661 23051 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d6869f01340579b37c459a66d3257 (DEFAULT_TABLE table=test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:32.990661 23185 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d6869f01340579b37c459a66d3257 (DEFAULT_TABLE table=test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:32.991570 23319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6d6869f01340579b37c459a66d3257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:32.991578 23051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6d6869f01340579b37c459a66d3257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:32.991578 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6d6869f01340579b37c459a66d3257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:32.997668 23402 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Bootstrap starting.
I20260619 11:12:32.998808 23403 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Bootstrap starting.
I20260619 11:12:32.999104 23404 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Bootstrap starting.
I20260619 11:12:33.000674 23402 tablet_bootstrap.cc:654] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:33.000813 23404 tablet_bootstrap.cc:654] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:33.001362 23403 tablet_bootstrap.cc:654] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:33.001655 23402 log.cc:824] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:33.001945 23404 log.cc:824] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:33.002148 23403 log.cc:824] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:33.003692 23402 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: No bootstrap required, opened a new log
I20260619 11:12:33.003688 23404 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: No bootstrap required, opened a new log
I20260619 11:12:33.003722 23403 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: No bootstrap required, opened a new log
I20260619 11:12:33.003849 23403 ts_tablet_manager.cc:1397] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:12:33.003862 23404 ts_tablet_manager.cc:1397] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:12:33.003896 23402 ts_tablet_manager.cc:1397] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:12:33.006603 23403 raft_consensus.cc:348] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.006840 23403 raft_consensus.cc:374] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:33.006892 23403 raft_consensus.cc:729] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea35665b1c14b0f90b98b5edf8c82eb, State: Initialized, Role: FOLLOWER
I20260619 11:12:33.007321 23403 consensus_queue.cc:260] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.007563 23404 raft_consensus.cc:348] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.007563 23402 raft_consensus.cc:348] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.007808 23404 raft_consensus.cc:374] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:33.007808 23402 raft_consensus.cc:374] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:33.007865 23402 raft_consensus.cc:729] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcee575eb24eb4952f75ec57d25592, State: Initialized, Role: FOLLOWER
I20260619 11:12:33.007865 23404 raft_consensus.cc:729] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0eb28bca444418fac377b397394d86c, State: Initialized, Role: FOLLOWER
I20260619 11:12:33.008074 23251 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
I20260619 11:12:33.008275 23403 ts_tablet_manager.cc:1428] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:12:33.008309 23404 consensus_queue.cc:260] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.008309 23402 consensus_queue.cc:260] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.009128 23385 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
I20260619 11:12:33.009187 23402 ts_tablet_manager.cc:1428] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 11:12:33.009187 23404 ts_tablet_manager.cc:1428] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:12:33.010108 23117 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
W20260619 11:12:33.015142 23118 tablet.cc:2369] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:33.025698 23408 raft_consensus.cc:482] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:33.025841 23408 raft_consensus.cc:504] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.026670 23408 leader_election.cc:288] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), a0eb28bca444418fac377b397394d86c (127.7.90.65:37447)
I20260619 11:12:33.030309 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.026956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:48801 (local address 127.7.90.65:37447)
0619 11:12:33.027103 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:33.027107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:33.027128 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:12:33.027286 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:33.027289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:33.027337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:33.027415 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:33.027731 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.028390 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.029396 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.029791 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.029901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:33.029956 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:33.030144 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 11:12:33.030206 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 11:12:33.030311 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.026879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.7.90.66:48801)
0619 11:12:33.027072 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:33.027088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:33.027170 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:33.027439 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:33.027441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:33.027453 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:33.027598 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:33.027608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.028492 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.028504 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:33.029273 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:33.029282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.029899 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.029903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:33.030004 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:33.030045 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 11:12:33.030130 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:12:33.030199 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":27}
I20260619 11:12:33.030445 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.026950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:44947 (local address 127.7.90.67:36949)
0619 11:12:33.027071 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:33.027075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:33.027292 (+ 217us) server_negotiation.cc:407] Connection header received
0619 11:12:33.027368 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:33.027373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:33.027421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:33.027511 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:33.027772 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.028421 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.029502 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.029900 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.030013 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:33.030077 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:33.030260 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:12:33.030321 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:12:33.030458 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.026879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:36949 (local address 127.7.90.66:44947)
0619 11:12:33.027204 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:33.027218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:33.027305 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:33.027512 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:33.027515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:33.027526 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:33.027630 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:33.027638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.028528 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.028533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:33.029386 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:33.029396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.030017 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.030020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:33.030145 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:33.030162 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:12:33.030247 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:12:33.030312 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":68,"thread_start_us":107,"threads_started":1}
I20260619 11:12:33.030900 23071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6d6869f01340579b37c459a66d3257" candidate_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0eb28bca444418fac377b397394d86c" is_pre_election: true
I20260619 11:12:33.031174 23071 raft_consensus.cc:2454] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea35665b1c14b0f90b98b5edf8c82eb in term 0.
I20260619 11:12:33.031154 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6d6869f01340579b37c459a66d3257" candidate_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bcee575eb24eb4952f75ec57d25592" is_pre_election: true
I20260619 11:12:33.031353 23339 raft_consensus.cc:2454] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea35665b1c14b0f90b98b5edf8c82eb in term 0.
I20260619 11:12:33.031584 23141 leader_election.cc:302] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea35665b1c14b0f90b98b5edf8c82eb, a0eb28bca444418fac377b397394d86c; no voters:
I20260619 11:12:33.031839 23408 raft_consensus.cc:2781] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:33.031910 23408 raft_consensus.cc:482] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:33.031939 23408 raft_consensus.cc:3037] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:33.032884 23408 raft_consensus.cc:504] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.033233 23408 leader_election.cc:288] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [CANDIDATE]: Term 1 election: Requested vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), a0eb28bca444418fac377b397394d86c (127.7.90.65:37447)
I20260619 11:12:33.033483 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6d6869f01340579b37c459a66d3257" candidate_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bcee575eb24eb4952f75ec57d25592"
I20260619 11:12:33.033639 23339 raft_consensus.cc:3037] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:33.033638 23071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6d6869f01340579b37c459a66d3257" candidate_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0eb28bca444418fac377b397394d86c"
I20260619 11:12:33.033756 23071 raft_consensus.cc:3037] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:33.034847 23339 raft_consensus.cc:2454] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea35665b1c14b0f90b98b5edf8c82eb in term 1.
I20260619 11:12:33.034847 23071 raft_consensus.cc:2454] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea35665b1c14b0f90b98b5edf8c82eb in term 1.
I20260619 11:12:33.035144 23139 leader_election.cc:302] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb; no voters:
I20260619 11:12:33.035296 23408 raft_consensus.cc:2781] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:33.035521 23408 raft_consensus.cc:686] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 LEADER]: Becoming Leader. State: Replica: 5ea35665b1c14b0f90b98b5edf8c82eb, State: Running, Role: LEADER
I20260619 11:12:33.035808 23408 consensus_queue.cc:237] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:33.038641 22925 catalog_manager.cc:5495] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66). New cstate: current_term: 1 leader_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } health_report { overall_health: HEALTHY } } }
I20260619 11:12:33.064072 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.060192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.7.90.66:37767)
0619 11:12:33.060311 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:33.060315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:33.060404 (+ 89us) server_negotiation.cc:407] Connection header received
0619 11:12:33.060598 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:33.060601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:33.060683 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:33.060805 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:33.061061 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.061726 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.062601 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:33.062854 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:33.062975 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:33.063270 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:33.063663 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:33.063792 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:33.063854 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 11:12:33.063905 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:12:33.069319 23071 raft_consensus.cc:1261] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Refusing update from remote peer 5ea35665b1c14b0f90b98b5edf8c82eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:33.069320 23339 raft_consensus.cc:1261] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Refusing update from remote peer 5ea35665b1c14b0f90b98b5edf8c82eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:33.070040 23412 consensus_queue.cc:1035] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:33.070225 23408 consensus_queue.cc:1035] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:33.080274 23416 mvcc.cc:205] Tried to move back new op lower bound from 7298529497365299200 to 7298529497237250048. Current Snapshot: MvccSnapshot[applied={T|T < 7298529497365299200 or (T in {7298529497365299200})}]
I20260619 11:12:33.085004 22925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:33.085170 22925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:33.087327 22925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Sending DeleteTablet for 3 replicas of tablet 4a6d6869f01340579b37c459a66d3257
I20260619 11:12:33.088029 23051 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6d6869f01340579b37c459a66d3257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:33 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:52926
I20260619 11:12:33.088125 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22988
I20260619 11:12:33.088215 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6d6869f01340579b37c459a66d3257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:33 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44264
I20260619 11:12:33.088594 23185 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6d6869f01340579b37c459a66d3257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:33 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:44338
I20260619 11:12:33.092592 23425 tablet_replica.cc:330] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: stopping tablet replica
I20260619 11:12:33.092904 23425 raft_consensus.cc:2229] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:12:33.093286 23425 raft_consensus.cc:2258] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:33.093704 23424 tablet_replica.cc:330] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: stopping tablet replica
I20260619 11:12:33.093982 23424 raft_consensus.cc:2229] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:33.094293 23424 raft_consensus.cc:2258] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:12:33.095033 22913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:37447 (error 108)
I20260619 11:12:33.095255 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23122
I20260619 11:12:33.096722 23424 ts_tablet_manager.cc:1905] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:33.096794 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.096228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.0.0.1:52928)
0619 11:12:33.096602 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:33.096691 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260619 11:12:33.097383 22913 catalog_manager.cc:4584] TS a0eb28bca444418fac377b397394d86c (127.7.90.65:37447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a6d6869f01340579b37c459a66d3257: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37447: connect: Connection refused (error 111)
I20260619 11:12:33.099190 23424 ts_tablet_manager.cc:1918] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:33.099291 23424 log.cc:1196] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/4a6d6869f01340579b37c459a66d3257
I20260619 11:12:33.099624 23424 ts_tablet_manager.cc:1939] T 4a6d6869f01340579b37c459a66d3257 P 47bcee575eb24eb4952f75ec57d25592: Deleting consensus metadata
I20260619 11:12:33.100925 22911 catalog_manager.cc:4841] TS 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): tablet 4a6d6869f01340579b37c459a66d3257 (table test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]) successfully deleted
I20260619 11:12:33.102349 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23256
I20260619 11:12:33.103442 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:33.102627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:37767 (local address 127.0.0.1:44362)
0619 11:12:33.103274 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:33.103338 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
I20260619 11:12:33.108747 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 22894
I20260619 11:12:53.115137 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--webserver_interface=127.7.90.126
--webserver_port=33945
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:45413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:12:53.223394 23427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:53.223666 23427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:53.223737 23427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:53.227586 23427 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:12:53.227653 23427 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:12:53.227680 23427 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:12:53.227708 23427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:12:53.227749 23427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:53.227790 23427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:53.227839 23427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:12:53.227874 23427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:12:53.232816 23427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:45413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:53.233867 23427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:53.234891 23427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:53.247462 23433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.247522 23427 server_base.cc:1034] running on GCE node
W20260619 11:12:53.247473 23434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.247484 23436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.248293 23427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:53.249394 23427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:53.250579 23427 hybrid_clock.cc:648] HybridClock initialized: now 1781867573250566 us; error 29 us; skew 500 ppm
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:53.257494 23427 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:12:53.258512 23427 webserver.cc:460] Webserver started at http://127.7.90.126:33945/ using document root <none> and password file <none>
I20260619 11:12:53.259124 23427 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:53.259238 23427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:53.263155 23427 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:12:53.265259 23443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.266526 23427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:53.266681 23427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2a5c29ed4d5f442999fc2b5077e1fa40"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "23397000aa53455434c38159c6ddf905"
server_key_iv: "1636c0b536bb2563bc818d96e3a73b39"
server_key_version: "encryptionkey@0"
I20260619 11:12:53.267242 23427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:53.284801 23427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:53.288067 23427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:53.288276 23427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:53.295212 23427 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:45413
I20260619 11:12:53.295477 23495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:45413 every 8 connection(s)
I20260619 11:12:53.296264 23427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:12:53.300385 23496 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:12:53.301148 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23427
I20260619 11:12:53.301568 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:37447
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=39527
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:53.302153 23496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap starting.
I20260619 11:12:53.324998 23496 log.cc:824] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:53.335736 23496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:53.336146 23496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap complete.
I20260619 11:12:53.341364 23496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:53.341861 23496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Initialized, Role: FOLLOWER
I20260619 11:12:53.342365 23496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:53.342511 23496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:12:53.342581 23496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:12:53.342669 23496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:12:53.345125 23496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:53.345501 23496 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5c29ed4d5f442999fc2b5077e1fa40; no voters:
I20260619 11:12:53.345865 23496 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:12:53.345999 23500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:12:53.346316 23500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 LEADER]: Becoming Leader. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Running, Role: LEADER
I20260619 11:12:53.346673 23500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:12:53.347110 23496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:12:53.348153 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5c29ed4d5f442999fc2b5077e1fa40. Latest consensus state: current_term: 2 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:12:53.348129 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:12:53.348268 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:53.348268 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:12:53.349220 23512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:12:53.353190 23512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:12:53.353776 23512 catalog_manager.cc:1259] Loaded cluster ID: 797b6955bfb54d6da95b69c06d2b1395
I20260619 11:12:53.353847 23512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:12:53.355530 23512 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:12:53.356086 23512 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Loaded TSK: 0
I20260619 11:12:53.384357 23512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Generated new TSK 1
I20260619 11:12:53.384725 23512 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:12:53.449069 23498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:53.449376 23498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:53.449460 23498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:53.453332 23498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:53.453414 23498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:53.453503 23498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:12:53.458755 23498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:37447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=39527
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:53.459820 23498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:53.460778 23498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:53.471019 23523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.471052 23524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.471061 23526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.471503 23498 server_base.cc:1034] running on GCE node
I20260619 11:12:53.471879 23498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:53.472470 23498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:53.473642 23498 hybrid_clock.cc:648] HybridClock initialized: now 1781867573473619 us; error 37 us; skew 500 ppm
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:53.476378 23498 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:12:53.477232 23498 webserver.cc:460] Webserver started at http://127.7.90.65:39527/ using document root <none> and password file <none>
I20260619 11:12:53.477891 23498 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:53.477993 23498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:53.481788 23498 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:12:53.483853 23533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.485119 23498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:53.485255 23498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "a0eb28bca444418fac377b397394d86c"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "2128b76be74dcab6ce9622451a9b7ab3"
server_key_iv: "cf3980ebaac52626423d4c21d844ebd1"
server_key_version: "encryptionkey@0"
I20260619 11:12:53.485841 23498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:53.523320 23498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:53.526180 23498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:53.526391 23498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:53.526999 23498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:53.528054 23540 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:12:53.532088 23498 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:12:53.532174 23498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:12:53.532258 23498 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:12:53.533921 23498 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:12:53.533991 23498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:53.534319 23540 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Bootstrap starting.
I20260619 11:12:53.544627 23498 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:37447
I20260619 11:12:53.544700 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:37447 every 8 connection(s)
I20260619 11:12:53.545619 23498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:12:53.548952 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23498
I20260619 11:12:53.549430 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:37767
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=46517
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:53.553876 23540 log.cc:824] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:53.561450 23540 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:53.561916 23540 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Bootstrap complete.
I20260619 11:12:53.563386 23540 ts_tablet_manager.cc:1397] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20260619 11:12:53.566169 23540 raft_consensus.cc:348] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:53.566677 23540 raft_consensus.cc:729] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0eb28bca444418fac377b397394d86c, State: Initialized, Role: FOLLOWER
I20260619 11:12:53.567107 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.547742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:55583 (local address 127.7.90.126:45413)
0619 11:12:53.548433 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.548453 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.551086 (+ 2633us) server_negotiation.cc:407] Connection header received
0619 11:12:53.551408 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.551433 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.551918 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.552292 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.553909 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.555156 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.557073 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.557749 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.557919 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.561235 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.561241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.561256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.561289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.563643 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.564249 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.564255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.564260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.564400 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.564761 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.564764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.564766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.565220 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.565352 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.565983 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.566274 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":153,"threads_started":1}
I20260619 11:12:53.567188 23540 consensus_queue.cc:260] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:53.567638 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.547732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:55583)
0619 11:12:53.548530 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.548603 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.549611 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:53.552391 (+ 2780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:53.552403 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:53.552922 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:53.553699 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.553727 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.555334 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.555342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.556927 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.556942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.557912 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.557918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.558169 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.558916 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:53.558942 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:53.561022 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:53.563777 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.563783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.563796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.564126 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.564499 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.564502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.564504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.564663 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.565336 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:53.565342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:53.565653 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 11:12:53.565844 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 11:12:53.566153 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260619 11:12:53.568153 23540 ts_tablet_manager.cc:1428] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:12:53.570078 23648 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:53.570304 23648 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:53.570807 23648 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
I20260619 11:12:53.573158 23460 ts_manager.cc:194] Registered new tserver with Master: a0eb28bca444418fac377b397394d86c (127.7.90.65:37447)
I20260619 11:12:53.576537 23460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:55583
I20260619 11:12:53.578845 23648 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.65@KRBTEST.COM
I20260619 11:12:53.586118 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.575968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.7.90.65:37447)
0619 11:12:53.576330 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.576340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.576880 (+ 540us) server_negotiation.cc:407] Connection header received
0619 11:12:53.576947 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.576952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.577139 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.577262 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.578102 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.579079 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.580021 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.580489 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.580614 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.582699 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.582708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.582719 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.582742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.584239 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.584693 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.584699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.584701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.584779 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.585073 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.585076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.585078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.585305 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.585441 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.585825 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.585978 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":102,"threads_started":1}
I20260619 11:12:53.586115 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.575874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.0.0.1:53292)
0619 11:12:53.576591 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.576634 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.576759 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:53.577421 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:53.577427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:53.577847 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:53.577973 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.577981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.579208 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.579211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.579899 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.579909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.580612 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.580616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.580734 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.581238 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:53.581248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:53.582569 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:53.584355 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.584365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.584371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.584583 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.584875 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.584878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.584880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.584958 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.585415 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:53.585421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:53.585589 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 11:12:53.585736 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 11:12:53.585936 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":108,"threads_started":1}
I20260619 11:12:53.587088 23582 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6d6869f01340579b37c459a66d3257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:33 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:53292
I20260619 11:12:53.588088 23657 tablet_replica.cc:330] stopping tablet replica
I20260619 11:12:53.588373 23657 raft_consensus.cc:2229] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:53.588608 23657 raft_consensus.cc:2258] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:53.589641 23657 ts_tablet_manager.cc:1905] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:53.593379 23657 ts_tablet_manager.cc:1918] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:53.593515 23657 log.cc:1196] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal/wals/4a6d6869f01340579b37c459a66d3257
I20260619 11:12:53.593854 23657 ts_tablet_manager.cc:1939] T 4a6d6869f01340579b37c459a66d3257 P a0eb28bca444418fac377b397394d86c: Deleting consensus metadata
I20260619 11:12:53.595444 23447 catalog_manager.cc:4841] TS a0eb28bca444418fac377b397394d86c (127.7.90.65:37447): tablet 4a6d6869f01340579b37c459a66d3257 (table test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]) successfully deleted
W20260619 11:12:53.665974 23652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:53.666313 23652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:53.666373 23652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:53.670324 23652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:53.670374 23652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:53.670435 23652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:12:53.675524 23652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:37767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=46517
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:53.676450 23652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:53.677390 23652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:53.686625 23665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.686581 23662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.686581 23663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.686640 23652 server_base.cc:1034] running on GCE node
I20260619 11:12:53.687244 23652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:53.687803 23652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:53.688977 23652 hybrid_clock.cc:648] HybridClock initialized: now 1781867573688941 us; error 57 us; skew 500 ppm
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:53.691588 23652 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:12:53.692420 23652 webserver.cc:460] Webserver started at http://127.7.90.66:46517/ using document root <none> and password file <none>
I20260619 11:12:53.692940 23652 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:53.693041 23652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:53.696540 23652 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:53.698544 23672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.699656 23652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:12:53.699787 23652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "c40e8a487390d80c04ece06ec047593b"
server_key_iv: "b0c5d12e20f10f6e8ec5911760a8c78f"
server_key_version: "encryptionkey@0"
I20260619 11:12:53.700311 23652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:53.741364 23652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:53.744207 23652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:53.744408 23652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:53.745069 23652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:53.746138 23679 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:12:53.750278 23652 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:12:53.750347 23652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:12:53.750450 23652 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:12:53.752007 23652 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:12:53.752072 23652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:12:53.752301 23679 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Bootstrap starting.
I20260619 11:12:53.757771 23679 log.cc:824] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:53.762773 23652 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:37767
I20260619 11:12:53.762832 23787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:37767 every 8 connection(s)
I20260619 11:12:53.763602 23652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:12:53.765965 23679 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:12:53.766384 23679 tablet_bootstrap.cc:492] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Bootstrap complete.
I20260619 11:12:53.766549 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23652
I20260619 11:12:53.766711 23679 ts_tablet_manager.cc:1397] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260619 11:12:53.766975 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:36949
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=39155
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:12:53.769500 23679 raft_consensus.cc:348] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:53.770143 23679 raft_consensus.cc:729] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea35665b1c14b0f90b98b5edf8c82eb, State: Initialized, Role: FOLLOWER
I20260619 11:12:53.770627 23679 consensus_queue.cc:260] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:53.771700 23679 ts_tablet_manager.cc:1428] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:53.778779 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.765355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:40755 (local address 127.7.90.126:45413)
0619 11:12:53.765505 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.765510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.766179 (+ 669us) server_negotiation.cc:407] Connection header received
0619 11:12:53.767337 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.767343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.767421 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.767516 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.768936 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.769780 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.771274 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.771548 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.771694 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.774879 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.774882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.774884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.774912 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.776370 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.777018 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.777020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.777022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.777092 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.777484 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.777486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.777487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.777823 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.777927 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.778554 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.778659 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:12:53.779816 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.765504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:40755)
0619 11:12:53.766014 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.766075 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.767075 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:53.767677 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:53.767685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:53.768149 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:53.768762 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.768779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.769927 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.769930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.771165 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.771175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.771694 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.771698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.771906 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.772808 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:53.772824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:53.774672 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:53.776489 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.776499 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.776516 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.776904 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.777186 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.777191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.777197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.777387 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.777940 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:53.777947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:53.778297 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 11:12:53.778532 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 11:12:53.778887 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260619 11:12:53.780886 23788 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:53.781111 23788 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:53.781634 23788 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
I20260619 11:12:53.783360 23460 ts_manager.cc:194] Registered new tserver with Master: 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
I20260619 11:12:53.784497 23460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:40755
I20260619 11:12:53.786823 23788 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.66@KRBTEST.COM
I20260619 11:12:53.793402 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.784196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:37767 (local address 127.0.0.1:39312)
0619 11:12:53.784351 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.784365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.784430 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:53.785093 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:53.785097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:53.785284 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:53.785360 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.785366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.786400 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.786405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.786968 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.786984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.787797 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.787801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.787904 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.788472 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:53.788486 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:53.789946 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:53.791907 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.791911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.791913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.792170 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.792444 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.792447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.792449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.792538 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.792924 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:53.792928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:53.793023 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 11:12:53.793132 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:12:53.793259 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 11:12:53.793429 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.784317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.7.90.66:37767)
0619 11:12:53.784683 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.784693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.784715 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:12:53.784779 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.784785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.784979 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.785112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.785471 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.786273 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.787112 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.787646 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.787788 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.790066 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.790075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.790087 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.790109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.791753 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.792270 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.792274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.792275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.792356 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.792626 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.792628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.792629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.792830 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.792928 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.793183 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.793311 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":287,"thread_start_us":105,"threads_started":1}
I20260619 11:12:53.794373 23721 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6d6869f01340579b37c459a66d3257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:33 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:39312
I20260619 11:12:53.795249 23794 tablet_replica.cc:330] stopping tablet replica
I20260619 11:12:53.795560 23794 raft_consensus.cc:2229] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:53.795795 23794 raft_consensus.cc:2258] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:53.796779 23794 ts_tablet_manager.cc:1905] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:53.800248 23794 ts_tablet_manager.cc:1918] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:53.800329 23794 log.cc:1196] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/4a6d6869f01340579b37c459a66d3257
I20260619 11:12:53.800586 23794 ts_tablet_manager.cc:1939] T 4a6d6869f01340579b37c459a66d3257 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting consensus metadata
I20260619 11:12:53.802069 23445 catalog_manager.cc:4841] TS 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): tablet 4a6d6869f01340579b37c459a66d3257 (table test-table [id=ba916caf27b54d409a384cb8c4c9c7b8]) successfully deleted
W20260619 11:12:53.886240 23791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:12:53.886471 23791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:12:53.886514 23791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:12:53.890574 23791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:12:53.890651 23791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:12:53.890712 23791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:12:53.895931 23791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:36949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=39155
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:12:53.896840 23791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:12:53.897867 23791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:12:53.907339 23800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.907361 23791 server_base.cc:1034] running on GCE node
W20260619 11:12:53.907295 23799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:12:53.907487 23802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:12:53.907919 23791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:12:53.908485 23791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:12:53.909603 23791 hybrid_clock.cc:648] HybridClock initialized: now 1781867573909577 us; error 30 us; skew 500 ppm
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:12:53.912281 23791 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:12:53.913136 23791 webserver.cc:460] Webserver started at http://127.7.90.67:39155/ using document root <none> and password file <none>
I20260619 11:12:53.913761 23791 fs_manager.cc:359] Metadata directory not provided
I20260619 11:12:53.913874 23791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:12:53.917709 23791 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:12:53.919790 23809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.921034 23791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:12:53.921190 23791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "47bcee575eb24eb4952f75ec57d25592"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "e3baf15aadd18f29348ddfcf113ec39e"
server_key_iv: "19a54e742bf36bf58567f9ff35923c4c"
server_key_version: "encryptionkey@0"
I20260619 11:12:53.921777 23791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:12:53.936151 23791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:12:53.938982 23791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:12:53.939110 23791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:12:53.939572 23791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:12:53.940265 23791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:12:53.940308 23791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.940344 23791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:12:53.940359 23791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:12:53.948854 23791 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:36949
I20260619 11:12:53.948916 23922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:36949 every 8 connection(s)
I20260619 11:12:53.949966 23791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:12:53.953796 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23791
Jun 19 11:12:53 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867573, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:12:53.963196 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.954426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.7.90.126:45413)
0619 11:12:53.954638 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.954642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.954777 (+ 135us) server_negotiation.cc:407] Connection header received
0619 11:12:53.954879 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.954881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.954929 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.955021 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.955699 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.956274 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.957365 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.957680 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.957752 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.959264 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.959268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.959270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.959299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.961444 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.961961 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.961964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.961965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.962035 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.962320 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.962324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.962326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.962565 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.962753 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.962943 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.963037 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260619 11:12:53.963712 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.951656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:39363 (local address 127.7.90.126:45413)
0619 11:12:53.951781 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.951784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.952739 (+ 955us) server_negotiation.cc:407] Connection header received
0619 11:12:53.953676 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.953679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.953739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.953833 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.955179 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.955746 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.957118 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.957311 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.957424 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.960118 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.960121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.960122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.960153 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.961623 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.962170 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.962173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.962174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.962233 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.962559 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.962562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.962563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.962734 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.962823 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.963428 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.963547 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260619 11:12:53.964588 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.952016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:39363)
0619 11:12:53.952606 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.952644 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.953424 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:53.954002 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:53.954009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:53.954409 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:12:53.955064 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.955085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.955868 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.955871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.957009 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:53.957020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.957470 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.957473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:53.957684 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.958263 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:12:53.958287 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:12:53.959987 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:12:53.961725 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.961731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.961746 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.962062 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.962322 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:12:53.962327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:12:53.962329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:12:53.962469 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:12:53.962849 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:12:53.962856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:12:53.963145 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 11:12:53.963381 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 11:12:53.963809 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":102,"threads_started":1}
I20260619 11:12:53.965602 23923 heartbeater.cc:347] Connected to a master server at 127.7.90.126:45413
I20260619 11:12:53.965821 23923 heartbeater.cc:464] Registering TS with master...
I20260619 11:12:53.966260 23923 heartbeater.cc:511] Master 127.7.90.126:45413 requested a full tablet report, sending...
I20260619 11:12:53.967211 23460 ts_manager.cc:194] Registered new tserver with Master: 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949)
I20260619 11:12:53.967744 23460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:39363
I20260619 11:12:53.976596 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:12:53.980218 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.977177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.7.90.126:45413)
0619 11:12:53.977310 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.977315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.977467 (+ 152us) server_negotiation.cc:407] Connection header received
0619 11:12:53.977627 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.977629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.977717 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.977792 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:53.978000 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.978485 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.979272 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.979478 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.979548 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.979789 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:53.979913 (+ 124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 11:12:53.980099 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43016: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 11:12:53.980283 23650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43016: token expired
W20260619 11:12:53.980640 23391 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.90.126:45413)
I20260619 11:12:53.987478 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.980956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.7.90.126:45413)
0619 11:12:53.981063 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:53.981069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:53.981099 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:12:53.981233 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:53.981236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:53.981277 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:53.981358 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:12:53.982013 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.982484 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.983249 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:53.983462 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:53.983564 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:53.984757 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:12:53.984760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:12:53.984763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:12:53.984793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:12:53.985956 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.986349 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.986353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.986355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.986439 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:12:53.986706 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:12:53.986710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:12:53.986712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:12:53.986868 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:12:53.987004 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:53.987229 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 11:12:53.987335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:12:53.989866 23460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:12:53.993088 23460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:12:54.000397 23721 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d454625404b388f95781e81c95022 (DEFAULT_TABLE table=test-table [id=4e53c405c23041ff8d587650d83d2cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:54.000727 23582 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d454625404b388f95781e81c95022 (DEFAULT_TABLE table=test-table [id=4e53c405c23041ff8d587650d83d2cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:54.001049 23721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d454625404b388f95781e81c95022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:54.001286 23582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d454625404b388f95781e81c95022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:54.003726 23679 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Bootstrap starting.
I20260619 11:12:54.004308 23540 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Bootstrap starting.
I20260619 11:12:54.004813 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.999564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:36949 (local address 127.0.0.1:45492)
0619 11:12:53.999872 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:53.999887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:53.999956 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:54.000516 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:54.000518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:54.000536 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:54.000625 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.000632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.002017 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.002020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.003203 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.003214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.004398 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.004402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.004507 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.004558 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 11:12:54.004614 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 11:12:54.004691 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 11:12:54.004817 23679 tablet_bootstrap.cc:654] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:54.005141 23540 tablet_bootstrap.cc:654] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:54.005543 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:53.999733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.7.90.67:36949)
0619 11:12:54.000013 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:54.000025 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:54.000049 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:12:54.000127 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:54.000133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:54.000354 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:54.000517 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:54.000739 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.001870 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.003382 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.004249 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.005015 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.005189 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:54.005244 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 11:12:54.005340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260619 11:12:54.006296 23679 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: No bootstrap required, opened a new log
I20260619 11:12:54.006385 23679 ts_tablet_manager.cc:1397] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:12:54.006744 23679 raft_consensus.cc:348] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.006842 23679 raft_consensus.cc:374] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:54.006868 23679 raft_consensus.cc:729] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea35665b1c14b0f90b98b5edf8c82eb, State: Initialized, Role: FOLLOWER
I20260619 11:12:54.006987 23679 consensus_queue.cc:260] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.007256 23679 ts_tablet_manager.cc:1428] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:12:54.008600 23857 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d454625404b388f95781e81c95022 (DEFAULT_TABLE table=test-table [id=4e53c405c23041ff8d587650d83d2cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:12:54.009724 23857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d454625404b388f95781e81c95022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:12:54.010105 23540 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: No bootstrap required, opened a new log
I20260619 11:12:54.010215 23540 ts_tablet_manager.cc:1397] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 11:12:54.010586 23540 raft_consensus.cc:348] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.010705 23540 raft_consensus.cc:374] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:54.010756 23540 raft_consensus.cc:729] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0eb28bca444418fac377b397394d86c, State: Initialized, Role: FOLLOWER
I20260619 11:12:54.010890 23540 consensus_queue.cc:260] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.011145 23540 ts_tablet_manager.cc:1428] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260619 11:12:54.014496 23789 tablet.cc:2369] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:54.017216 23935 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Bootstrap starting.
I20260619 11:12:54.019227 23935 tablet_bootstrap.cc:654] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Neither blocks nor log segments found. Creating new log.
I20260619 11:12:54.020434 23935 log.cc:824] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Log is configured to *not* fsync() on all Append() calls
I20260619 11:12:54.022248 23935 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: No bootstrap required, opened a new log
I20260619 11:12:54.022447 23935 ts_tablet_manager.cc:1397] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:12:54.025628 23935 raft_consensus.cc:348] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.025862 23935 raft_consensus.cc:374] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:12:54.025941 23935 raft_consensus.cc:729] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcee575eb24eb4952f75ec57d25592, State: Initialized, Role: FOLLOWER
I20260619 11:12:54.026369 23935 consensus_queue.cc:260] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.026943 23923 heartbeater.cc:503] Master 127.7.90.126:45413 was elected leader, sending a full tablet report...
I20260619 11:12:54.027256 23935 ts_tablet_manager.cc:1428] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:12:54.046697 23649 tablet.cc:2369] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:12:54.200846 23924 tablet.cc:2369] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:12:54.255064 23937 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:12:54.255257 23937 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.256328 23937 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0eb28bca444418fac377b397394d86c (127.7.90.65:37447), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
I20260619 11:12:54.260157 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:54.256638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:44301 (local address 127.7.90.65:37447)
0619 11:12:54.256920 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:54.256924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:54.256936 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:12:54.256989 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:54.256991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:54.257043 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:54.257165 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:54.257427 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.258021 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.259028 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.259574 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.259717 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.259913 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:54.259955 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 11:12:54.260021 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260619 11:12:54.260154 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:54.256545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.7.90.67:44301)
0619 11:12:54.256662 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:54.256677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:54.256774 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:54.257177 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:54.257180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:54.257198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:54.257315 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.257320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.258137 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.258141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.258896 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.258904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.259716 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.259719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.259849 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.259870 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:12:54.259938 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:12:54.260023 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 11:12:54.260465 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:54.256654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:57697 (local address 127.7.90.66:37767)
0619 11:12:54.256795 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:54.256800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:54.257227 (+ 427us) server_negotiation.cc:407] Connection header received
0619 11:12:54.257369 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:54.257376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:54.257431 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:54.257523 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:12:54.257838 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.258377 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.259499 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.259944 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.260063 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.260234 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:54.260271 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 11:12:54.260338 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:12:54.260460 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:54.256551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:37767 (local address 127.7.90.67:57697)
0619 11:12:54.257129 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:54.257141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:12:54.257251 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:12:54.257561 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:12:54.257564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:12:54.257578 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:12:54.257722 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.257727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.258486 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.258492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.259323 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:12:54.259332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.260061 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.260065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:12:54.260157 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.260179 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:12:54.260242 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:12:54.260310 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":250,"thread_start_us":137,"threads_started":1}
I20260619 11:12:54.261085 23742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d454625404b388f95781e81c95022" candidate_uuid: "47bcee575eb24eb4952f75ec57d25592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" is_pre_election: true
I20260619 11:12:54.261081 23602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d454625404b388f95781e81c95022" candidate_uuid: "47bcee575eb24eb4952f75ec57d25592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0eb28bca444418fac377b397394d86c" is_pre_election: true
I20260619 11:12:54.261435 23602 raft_consensus.cc:2454] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bcee575eb24eb4952f75ec57d25592 in term 0.
I20260619 11:12:54.261435 23742 raft_consensus.cc:2454] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bcee575eb24eb4952f75ec57d25592 in term 0.
I20260619 11:12:54.261984 23813 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb, a0eb28bca444418fac377b397394d86c; no voters:
I20260619 11:12:54.262282 23937 raft_consensus.cc:2781] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:12:54.262364 23937 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:12:54.262415 23937 raft_consensus.cc:3037] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:54.263448 23937 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.263792 23937 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [CANDIDATE]: Term 1 election: Requested vote from peers a0eb28bca444418fac377b397394d86c (127.7.90.65:37447), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
I20260619 11:12:54.264132 23602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d454625404b388f95781e81c95022" candidate_uuid: "47bcee575eb24eb4952f75ec57d25592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0eb28bca444418fac377b397394d86c"
I20260619 11:12:54.264176 23742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d454625404b388f95781e81c95022" candidate_uuid: "47bcee575eb24eb4952f75ec57d25592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
I20260619 11:12:54.264271 23602 raft_consensus.cc:3037] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:54.264302 23742 raft_consensus.cc:3037] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:12:54.265282 23602 raft_consensus.cc:2454] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bcee575eb24eb4952f75ec57d25592 in term 1.
I20260619 11:12:54.265450 23742 raft_consensus.cc:2454] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bcee575eb24eb4952f75ec57d25592 in term 1.
I20260619 11:12:54.265625 23813 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bcee575eb24eb4952f75ec57d25592, a0eb28bca444418fac377b397394d86c; no voters:
I20260619 11:12:54.265838 23937 raft_consensus.cc:2781] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:12:54.266108 23937 raft_consensus.cc:686] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 1 LEADER]: Becoming Leader. State: Replica: 47bcee575eb24eb4952f75ec57d25592, State: Running, Role: LEADER
I20260619 11:12:54.266480 23937 consensus_queue.cc:237] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:12:54.269503 23460 catalog_manager.cc:5495] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67). New cstate: current_term: 1 leader_uuid: "47bcee575eb24eb4952f75ec57d25592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } health_report { overall_health: UNKNOWN } } }
I20260619 11:12:54.305176 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:12:54.301508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.7.90.67:36949)
0619 11:12:54.301673 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 11:12:54.301678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:12:54.301692 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:12:54.301855 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:12:54.301857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:12:54.301912 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:12:54.302022 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:12:54.302276 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.302788 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.303713 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:12:54.303944 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:12:54.304052 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:12:54.304344 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:12:54.304772 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:12:54.304899 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:12:54.304964 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 11:12:54.305018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 11:12:54.310689 23742 raft_consensus.cc:1261] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Refusing update from remote peer 47bcee575eb24eb4952f75ec57d25592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:54.310689 23602 raft_consensus.cc:1261] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Refusing update from remote peer 47bcee575eb24eb4952f75ec57d25592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:12:54.311357 23937 consensus_queue.cc:1035] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:54.311537 23940 consensus_queue.cc:1035] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:12:54.317059 23943 mvcc.cc:205] Tried to move back new op lower bound from 7298529584369143808 to 7298529584197865472. Current Snapshot: MvccSnapshot[applied={T|T < 7298529584369143808}]
I20260619 11:12:54.317963 23944 mvcc.cc:205] Tried to move back new op lower bound from 7298529584369143808 to 7298529584197865472. Current Snapshot: MvccSnapshot[applied={T|T < 7298529584369143808}]
I20260619 11:12:54.319391 23945 mvcc.cc:205] Tried to move back new op lower bound from 7298529584369143808 to 7298529584197865472. Current Snapshot: MvccSnapshot[applied={T|T < 7298529584369143808 or (T in {7298529584369143808})}]
I20260619 11:12:54.325700 23460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:54.325827 23460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:12:54.328032 23460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Sending DeleteTablet for 3 replicas of tablet bb5d454625404b388f95781e81c95022
I20260619 11:12:54.328627 23582 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d454625404b388f95781e81c95022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:54 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:53292
I20260619 11:12:54.328655 23721 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d454625404b388f95781e81c95022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:54 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:39312
I20260619 11:12:54.328809 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23498
I20260619 11:12:54.329478 23953 tablet_replica.cc:330] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: stopping tablet replica
I20260619 11:12:54.329674 23953 raft_consensus.cc:2229] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:12:54.329807 23953 raft_consensus.cc:2258] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:12:54.330639 23953 ts_tablet_manager.cc:1905] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:54.333240 23953 ts_tablet_manager.cc:1918] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:54.333326 23953 log.cc:1196] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/bb5d454625404b388f95781e81c95022
I20260619 11:12:54.333680 23953 ts_tablet_manager.cc:1939] T bb5d454625404b388f95781e81c95022 P 5ea35665b1c14b0f90b98b5edf8c82eb: Deleting consensus metadata
I20260619 11:12:54.334368 23445 catalog_manager.cc:4841] TS 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): tablet bb5d454625404b388f95781e81c95022 (table test-table [id=4e53c405c23041ff8d587650d83d2cfb]) successfully deleted
I20260619 11:12:54.334390 23857 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d454625404b388f95781e81c95022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:12:54 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:45492
I20260619 11:12:54.334899 23954 tablet_replica.cc:330] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: stopping tablet replica
I20260619 11:12:54.335232 23954 raft_consensus.cc:2229] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 1 LEADER]: Raft consensus shutting down.
W20260619 11:12:54.335608 23447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:37447 (error 108)
I20260619 11:12:54.335657 23954 raft_consensus.cc:2258] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:12:54.336835 23655 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:12:54.336533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:37447 (local address 127.0.0.1:53304)
0619 11:12:54.336660 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:12:54.336739 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37447: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:12:54.336998 23954 ts_tablet_manager.cc:1905] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:12:54.337039 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23652
W20260619 11:12:54.337093 23447 catalog_manager.cc:4584] TS a0eb28bca444418fac377b397394d86c (127.7.90.65:37447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb5d454625404b388f95781e81c95022: Network error: Client connection negotiation failed: client connection to 127.7.90.65:37447: connect: Connection reset by peer (error 104)
I20260619 11:12:54.340027 23954 ts_tablet_manager.cc:1918] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:12:54.340126 23954 log.cc:1196] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/bb5d454625404b388f95781e81c95022
I20260619 11:12:54.340483 23954 ts_tablet_manager.cc:1939] T bb5d454625404b388f95781e81c95022 P 47bcee575eb24eb4952f75ec57d25592: Deleting consensus metadata
I20260619 11:12:54.341609 23445 catalog_manager.cc:4841] TS 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): tablet bb5d454625404b388f95781e81c95022 (table test-table [id=4e53c405c23041ff8d587650d83d2cfb]) successfully deleted
I20260619 11:12:54.344280 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23791
I20260619 11:12:54.349366 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 23427
I20260619 11:13:14.356185 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--webserver_interface=127.7.90.126
--webserver_port=33945
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:45413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:14.465585 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:14.465924 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:14.465997 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:14.470148 23956 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:13:14.470218 23956 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:13:14.470243 23956 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:13:14.470268 23956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:14.470330 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:14.470371 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:14.470407 23956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:14.470441 23956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:14.475950 23956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:45413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:45413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:14.477032 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:14.478122 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:14.486719 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:14.486730 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:14.486891 23956 server_base.cc:1034] running on GCE node
W20260619 11:13:14.487025 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:14.487627 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:14.488690 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:14.489854 23956 hybrid_clock.cc:648] HybridClock initialized: now 1781867594489831 us; error 39 us; skew 500 ppm
Jun 19 11:13:14 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:14.492476 23956 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:14.493291 23956 webserver.cc:460] Webserver started at http://127.7.90.126:33945/ using document root <none> and password file <none>
I20260619 11:13:14.493901 23956 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:14.493963 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:14.497828 23956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:14.499773 23971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:14.500933 23956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:14.501068 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2a5c29ed4d5f442999fc2b5077e1fa40"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "23397000aa53455434c38159c6ddf905"
server_key_iv: "1636c0b536bb2563bc818d96e3a73b39"
server_key_version: "encryptionkey@0"
I20260619 11:13:14.501617 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:14.522464 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:14.525419 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:14.525640 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:14.532485 23956 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:45413
I20260619 11:13:14.532780 24023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:45413 every 8 connection(s)
I20260619 11:13:14.533643 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:14.538154 24024 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:13:14.540060 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap starting.
I20260619 11:13:14.542915 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 23956
I20260619 11:13:14.543376 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:37447
--local_ip_for_outbound_sockets=127.7.90.65
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=39527
--webserver_interface=127.7.90.65
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:13:14.563241 24024 log.cc:824] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:14.577126 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:13:14.577608 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: Bootstrap complete.
I20260619 11:13:14.582986 24024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:13:14.583493 24024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Initialized, Role: FOLLOWER
I20260619 11:13:14.584033 24024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:13:14.584184 24024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:14.584254 24024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:14.584355 24024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:13:14.586972 24024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:13:14.587343 24024 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5c29ed4d5f442999fc2b5077e1fa40; no voters:
I20260619 11:13:14.587674 24024 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:13:14.587838 24028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:13:14.588281 24028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [term 3 LEADER]: Becoming Leader. State: Replica: 2a5c29ed4d5f442999fc2b5077e1fa40, State: Running, Role: LEADER
I20260619 11:13:14.588632 24028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } }
I20260619 11:13:14.588917 24024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:14.590195 24029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:13:14.590354 24029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:14.590195 24030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5c29ed4d5f442999fc2b5077e1fa40. Latest consensus state: current_term: 3 leader_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5c29ed4d5f442999fc2b5077e1fa40" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 45413 } } }
I20260619 11:13:14.590473 24030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:14.594312 24043 catalog_manager.cc:1259] Loaded cluster ID: 797b6955bfb54d6da95b69c06d2b1395
I20260619 11:13:14.594376 24043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: loading cluster ID for follower catalog manager: success
I20260619 11:13:14.596896 24043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a5c29ed4d5f442999fc2b5077e1fa40: acquiring CA information for follower catalog manager: success
F20260619 11:13:14.598208 24043 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6f683469ad google::LogMessage::Fail() at ??:0
@ 0x7f6f6834988a google::LogMessage::SendToLog() at ??:0
@ 0x7f6f683464de google::LogMessage::Flush() at ??:0
@ 0x7f6f68347a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f6b00a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6f72cab989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f6f72c88023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f6f72c8854b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f6f72c81652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f6f72c809e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f6f72cc7541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6f6a589512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6f6bbcd6db start_thread at ??:0
@ 0x7f6f66e4461f clone at ??:0
W20260619 11:13:14.688763 24026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:14.689018 24026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:14.689064 24026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:14.693136 24026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:14.693189 24026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:14.693280 24026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:14.698514 24026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:37447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=39527
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:14.699525 24026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:14.700584 24026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:14.709430 24067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:14.709430 24065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:14.709530 24064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:14.710098 24026 server_base.cc:1034] running on GCE node
I20260619 11:13:14.710481 24026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:14.711091 24026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:14.712267 24026 hybrid_clock.cc:648] HybridClock initialized: now 1781867594712234 us; error 52 us; skew 500 ppm
Jun 19 11:13:14 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:14.714875 24026 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:13:14.715682 24026 webserver.cc:460] Webserver started at http://127.7.90.65:39527/ using document root <none> and password file <none>
I20260619 11:13:14.716188 24026 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:14.716245 24026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:14.719965 24026 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:13:14.722116 24074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:14.723382 24026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:14.723501 24026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "a0eb28bca444418fac377b397394d86c"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "2128b76be74dcab6ce9622451a9b7ab3"
server_key_iv: "cf3980ebaac52626423d4c21d844ebd1"
server_key_version: "encryptionkey@0"
I20260619 11:13:14.724014 24026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:14.739935 24026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:14.742888 24026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:14.743072 24026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:14.743608 24026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:14.744604 24081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:13:14.749732 24026 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:13:14.749816 24026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 11:13:14.749876 24026 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:13:14.751472 24026 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:13:14.751535 24026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:13:14.751823 24081 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Bootstrap starting.
I20260619 11:13:14.761921 24026 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:37447
I20260619 11:13:14.762863 24026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:14.763387 24188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:37447 every 8 connection(s)
I20260619 11:13:14.770691 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24026
I20260619 11:13:14.771121 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:37767
--local_ip_for_outbound_sockets=127.7.90.66
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=46517
--webserver_interface=127.7.90.66
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:13:14.773679 24081 log.cc:824] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:14.782383 24081 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:13:14.782791 24081 tablet_bootstrap.cc:492] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Bootstrap complete.
I20260619 11:13:14.784174 24081 ts_tablet_manager.cc:1397] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260619 11:13:14.787389 24081 raft_consensus.cc:348] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:14.787917 24081 raft_consensus.cc:729] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0eb28bca444418fac377b397394d86c, State: Initialized, Role: FOLLOWER
I20260619 11:13:14.788412 24081 consensus_queue.cc:260] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:14.789364 24081 ts_tablet_manager.cc:1428] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260619 11:13:14.808430 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:13:14.764847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:56583)
0619 11:13:14.765298 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.765351 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:14.766176 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:14.807540 (+ 41364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
W20260619 11:13:14.808947 24189 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45413 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:45413: Client connection negotiation failed: client connection to 127.7.90.126:45413: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:13:14.809718 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:14.809401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:34757)
0619 11:13:14.809568 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.809645 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 11:13:14.810427 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:14.810225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:51525)
0619 11:13:14.810328 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.810374 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:13:14.810627 24189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:13:14.880522 24192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:14.880796 24192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:14.880877 24192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:14.884811 24192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:14.884886 24192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:14.884984 24192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:14.889993 24192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:37767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=46517
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:14.890940 24192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:14.891873 24192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:14.901574 24199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:14.901719 24200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:14.901713 24192 server_base.cc:1034] running on GCE node
W20260619 11:13:14.901582 24202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:14.902273 24192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:14.902889 24192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:14.904069 24192 hybrid_clock.cc:648] HybridClock initialized: now 1781867594904057 us; error 30 us; skew 500 ppm
Jun 19 11:13:14 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:14.906678 24192 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:13:14.907465 24192 webserver.cc:460] Webserver started at http://127.7.90.66:46517/ using document root <none> and password file <none>
I20260619 11:13:14.907975 24192 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:14.908057 24192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:14.911559 24192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:14.913595 24209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:14.914723 24192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:14.914845 24192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "5ea35665b1c14b0f90b98b5edf8c82eb"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "c40e8a487390d80c04ece06ec047593b"
server_key_iv: "b0c5d12e20f10f6e8ec5911760a8c78f"
server_key_version: "encryptionkey@0"
I20260619 11:13:14.915344 24192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:14.959549 24192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:14.962507 24192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:14.962700 24192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:14.963280 24192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:14.964067 24192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:14.964133 24192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:14.964198 24192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:14.964243 24192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:14.972967 24192 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:37767
I20260619 11:13:14.972986 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:37767 every 8 connection(s)
I20260619 11:13:14.973991 24192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:14.977303 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:14.975951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:50517)
0619 11:13:14.976553 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.976651 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":100,"threads_started":1}
I20260619 11:13:14.977608 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24192
W20260619 11:13:14.977784 24323 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45413 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:45413: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
I20260619 11:13:14.978055 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:36949
--local_ip_for_outbound_sockets=127.7.90.67
--tserver_master_addrs=127.7.90.126:45413
--webserver_port=39155
--webserver_interface=127.7.90.67
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:13:14.978300 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:14.978088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:38099)
0619 11:13:14.978194 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.978241 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 11:13:14.978897 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:14.978699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:42059)
0619 11:13:14.978797 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:14.978840 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 11:13:14.979094 24323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:13:15.084571 24326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:15.084790 24326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:15.084838 24326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:15.088670 24326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:15.088717 24326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:15.088805 24326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:15.093779 24326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:36949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=39155
--tserver_master_addrs=127.7.90.126:45413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:15.094672 24326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:15.095613 24326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:15.104611 24334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:15.104672 24331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:15.104686 24332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:15.105003 24326 server_base.cc:1034] running on GCE node
I20260619 11:13:15.105358 24326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:15.105989 24326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:15.107165 24326 hybrid_clock.cc:648] HybridClock initialized: now 1781867595107134 us; error 42 us; skew 500 ppm
Jun 19 11:13:15 dist-test-slave-f1t5 krb5kdc[22881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867595, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:15.109763 24326 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:13:15.110565 24326 webserver.cc:460] Webserver started at http://127.7.90.67:39155/ using document root <none> and password file <none>
I20260619 11:13:15.111048 24326 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:15.111101 24326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:15.114650 24326 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:15.116641 24341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:15.117911 24326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:15.118008 24326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "47bcee575eb24eb4952f75ec57d25592"
format_stamp: "Formatted at 2026-06-19 11:12:32 on dist-test-slave-f1t5"
server_key: "e3baf15aadd18f29348ddfcf113ec39e"
server_key_iv: "19a54e742bf36bf58567f9ff35923c4c"
server_key_version: "encryptionkey@0"
I20260619 11:13:15.118479 24326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:15.150738 24326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:15.153465 24326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:15.153636 24326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:15.154247 24326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:15.155043 24326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:15.155088 24326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:15.155165 24326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:15.155201 24326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:15.164091 24326 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:36949
I20260619 11:13:15.164120 24454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:36949 every 8 connection(s)
I20260619 11:13:15.165040 24326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:15.168396 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:15.167006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:49971)
0619 11:13:15.167629 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:15.167728 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":112,"threads_started":1}
W20260619 11:13:15.168881 24455 heartbeater.cc:644] Failed to heartbeat to 127.7.90.126:45413 (0 consecutive failures): Network error: Failed to ping master at 127.7.90.126:45413: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
I20260619 11:13:15.169409 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:15.169220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:48435)
0619 11:13:15.169314 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:15.169358 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 11:13:15.169965 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:15.169769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:37991)
0619 11:13:15.169879 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:15.169917 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 11:13:15.170115 24455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:13:15.174222 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24326
I20260619 11:13:15.811533 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:15.811144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:56501)
0619 11:13:15.811370 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:15.811420 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260619 11:13:15.979884 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:15.979551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:43247)
0619 11:13:15.979756 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:15.979802 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260619 11:13:16.019061 24461 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:16.019254 24461 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:16.020439 24461 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
Jun 19 11:13:16 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.67@KRBTEST.COM
Jun 19 11:13:16 dist-test-slave-f1t5 krb5kdc[22881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.66@KRBTEST.COM
I20260619 11:13:16.033699 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.020601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:36949 (local address 127.7.90.65:41659)
0619 11:13:16.020718 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:16.020735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:16.020838 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:16.022903 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:16.022909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:16.023413 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:16.023887 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:16.023901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.025126 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.025129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:16.026077 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:16.026086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.026730 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:16.026890 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:16.027707 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:16.027718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:16.029146 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:16.031352 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:16.031358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:16.031371 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:16.031694 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:16.032062 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:16.032065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:16.032067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:16.032230 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:16.032818 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:16.032824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:16.033028 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 11:13:16.033218 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 11:13:16.033495 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":137}
I20260619 11:13:16.033874 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.020894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:37767 (local address 127.7.90.65:37715)
0619 11:13:16.021201 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:16.021214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:16.021306 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:16.022905 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:16.022910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:16.023415 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:16.023887 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:16.023901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.025097 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.025100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:16.026072 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:16.026081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.026728 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:16.026897 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:16.027653 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:16.027668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:16.029459 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:16.031472 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:16.031474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:16.031476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:16.031701 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:16.032611 (+ 910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:16.032615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:16.032617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:16.032693 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:16.033239 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:16.033242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:16.033341 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 11:13:16.033448 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:13:16.033654 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":121,"spinlock_wait_cycles":20864,"thread_start_us":74,"threads_started":1}
I20260619 11:13:16.034050 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.021063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:37715 (local address 127.7.90.66:37767)
0619 11:13:16.021445 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:16.021467 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:16.021507 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:13:16.022227 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:16.022237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:16.022554 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:16.022842 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:16.024035 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.024965 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026176 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.026617 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026780 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:16.029694 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:16.029699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:16.029709 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:16.029734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:16.031362 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:16.032378 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:16.032383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:16.032388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:16.032519 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:16.032786 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:16.032790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:16.032792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:16.033118 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:16.033266 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:16.033619 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 11:13:16.033883 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":70,"threads_started":1}
I20260619 11:13:16.034238 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.020778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:41659 (local address 127.7.90.67:36949)
0619 11:13:16.021161 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:16.021177 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:16.021220 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:13:16.022089 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:16.022096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:16.022503 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:16.022829 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:16.024043 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.024989 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026197 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:16.026616 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:16.026773 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:16.029367 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:16.029376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:16.029393 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:16.029425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:16.031238 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:16.031798 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:16.031804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:16.031809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:16.031957 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:16.032344 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:16.032347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:16.032349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:16.032707 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:16.032851 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:16.033321 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 11:13:16.033596 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
W20260619 11:13:16.036211 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
W20260619 11:13:16.036415 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
I20260619 11:13:16.036481 24076 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0eb28bca444418fac377b397394d86c; no voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb
I20260619 11:13:16.036723 24461 raft_consensus.cc:2726] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:13:16.171152 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.170688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:50457)
0619 11:13:16.170954 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:16.171026 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260619 11:13:16.812598 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.812200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:45845)
0619 11:13:16.812450 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:16.812502 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260619 11:13:16.981025 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:16.980629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:38741)
0619 11:13:16.980884 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:16.980942 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260619 11:13:17.172259 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:17.171866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:48561)
0619 11:13:17.172096 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:17.172148 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260619 11:13:17.760407 24469 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:17.760532 24469 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:17.760879 24469 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
W20260619 11:13:17.761760 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
W20260619 11:13:17.761907 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
I20260619 11:13:17.762002 24076 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0eb28bca444418fac377b397394d86c; no voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb
I20260619 11:13:17.762143 24469 raft_consensus.cc:2726] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:13:17.813699 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:17.813271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:43191)
0619 11:13:17.813484 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:17.813607 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260619 11:13:17.982146 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:17.981763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:35739)
0619 11:13:17.981992 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:17.982042 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260619 11:13:18.173367 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:18.172910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:58383)
0619 11:13:18.173229 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:18.173281 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":89,"threads_started":1}
I20260619 11:13:18.814859 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:18.814452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:43669)
0619 11:13:18.814680 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:18.814735 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260619 11:13:18.983314 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:18.982882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:41373)
0619 11:13:18.983141 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:18.983202 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260619 11:13:19.181262 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:19.180911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:45449)
0619 11:13:19.181128 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:19.181176 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260619 11:13:19.526064 24476 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:19.526218 24476 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:19.526701 24476 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
W20260619 11:13:19.527387 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
W20260619 11:13:19.527530 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
I20260619 11:13:19.527581 24076 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0eb28bca444418fac377b397394d86c; no voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb
I20260619 11:13:19.527701 24476 raft_consensus.cc:2726] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:13:19.816215 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:19.815785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:58161)
0619 11:13:19.816071 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:19.816123 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260619 11:13:19.984413 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:19.984025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:52719)
0619 11:13:19.984273 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:19.984326 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260619 11:13:20.182281 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:20.181916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:52377)
0619 11:13:20.182140 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:20.182193 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260619 11:13:20.817377 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:20.816970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:52725)
0619 11:13:20.817202 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:20.817257 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260619 11:13:20.985473 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:20.985112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:33311)
0619 11:13:20.985332 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:20.985386 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260619 11:13:21.070746 24482 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:21.070880 24482 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:21.071368 24482 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
W20260619 11:13:21.072163 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
W20260619 11:13:21.072268 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
I20260619 11:13:21.072311 24076 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0eb28bca444418fac377b397394d86c; no voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb
I20260619 11:13:21.072444 24482 raft_consensus.cc:2726] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:13:21.183446 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:21.183087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:59965)
0619 11:13:21.183312 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:21.183363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260619 11:13:21.818796 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:21.818287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:48897)
0619 11:13:21.818638 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:21.818704 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20260619 11:13:21.986833 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:21.986375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:42233)
0619 11:13:21.986682 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:21.986743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":68,"threads_started":1}
I20260619 11:13:22.184677 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:22.184227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:34967)
0619 11:13:22.184526 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:22.184587 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":75,"threads_started":1}
I20260619 11:13:22.819831 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:22.819449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:54815)
0619 11:13:22.819669 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:22.819737 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260619 11:13:22.988044 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:22.987610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:50133)
0619 11:13:22.987865 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:22.987924 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260619 11:13:23.185755 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:23.185317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:44509)
0619 11:13:23.185607 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:23.185661 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260619 11:13:23.820870 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:23.820484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:38123)
0619 11:13:23.820719 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:23.820772 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260619 11:13:23.989166 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:23.988796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:51739)
0619 11:13:23.989036 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:23.989084 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260619 11:13:24.186784 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:24.186389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:51581)
0619 11:13:24.186636 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:24.186690 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260619 11:13:24.822057 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:24.821692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:53527)
0619 11:13:24.821913 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:24.821964 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260619 11:13:24.990250 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:24.989915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:55411)
0619 11:13:24.990117 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:24.990166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260619 11:13:25.187858 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:25.187501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:53261)
0619 11:13:25.187717 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:25.187768 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260619 11:13:25.602084 24496 raft_consensus.cc:482] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:25.602240 24496 raft_consensus.cc:504] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47bcee575eb24eb4952f75ec57d25592" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36949 } } peers { permanent_uuid: "a0eb28bca444418fac377b397394d86c" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 37447 } } peers { permanent_uuid: "5ea35665b1c14b0f90b98b5edf8c82eb" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 37767 } }
I20260619 11:13:25.602641 24496 leader_election.cc:288] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949), 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767)
W20260619 11:13:25.603446 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47bcee575eb24eb4952f75ec57d25592 (127.7.90.67:36949): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
W20260619 11:13:25.603636 24076 leader_election.cc:341] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ea35665b1c14b0f90b98b5edf8c82eb (127.7.90.66:37767): Not found: Tablet not found: bb5d454625404b388f95781e81c95022
I20260619 11:13:25.603694 24076 leader_election.cc:302] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0eb28bca444418fac377b397394d86c; no voters: 47bcee575eb24eb4952f75ec57d25592, 5ea35665b1c14b0f90b98b5edf8c82eb
I20260619 11:13:25.603827 24496 raft_consensus.cc:2726] T bb5d454625404b388f95781e81c95022 P a0eb28bca444418fac377b397394d86c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:13:25.823264 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:25.822874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:51893)
0619 11:13:25.823120 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:25.823171 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260619 11:13:25.991341 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:25.990888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:36653)
0619 11:13:25.991213 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:25.991263 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":61,"threads_started":1}
I20260619 11:13:26.188936 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:26.188534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:59849)
0619 11:13:26.188788 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:26.188845 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260619 11:13:26.824306 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:26.823951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:37959)
0619 11:13:26.824167 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:26.824215 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260619 11:13:26.992594 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:26.992132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:37929)
0619 11:13:26.992408 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:26.992478 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260619 11:13:27.190165 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:27.189737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:47743)
0619 11:13:27.190020 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:27.190071 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":77,"threads_started":1}
I20260619 11:13:27.825651 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:27.825054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:52097)
0619 11:13:27.825424 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:27.825482 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20260619 11:13:27.993779 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:27.993277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:53213)
0619 11:13:27.993634 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:27.993687 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260619 11:13:28.191151 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:28.190802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:34475)
0619 11:13:28.191025 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:28.191070 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260619 11:13:28.826786 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:28.826418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:50857)
0619 11:13:28.826650 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:28.826698 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260619 11:13:28.994927 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:28.994526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:48543)
0619 11:13:28.994784 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:28.994840 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260619 11:13:29.192356 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:29.191877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.67:59837)
0619 11:13:29.192169 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:29.192249 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260619 11:13:29.828336 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:29.827843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.65:43193)
0619 11:13:29.828174 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:29.828236 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260619 11:13:29.996121 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:29.995748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:45413 (local address 127.7.90.66:52737)
0619 11:13:29.995991 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:29.996039 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.126:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260619 11:13:30.180075 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24026
I20260619 11:13:30.185755 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24192
I20260619 11:13:30.190768 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24326
2026-06-19T11:13:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59277 ms)
[----------] 30 tests from SecurityITest (234336 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:30 dist-test-slave-f1t5 krb5kdc[24513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:30 dist-test-slave-f1t5 krb5kdc[24513](info): set up 2 sockets
Jun 19 11:13:30 dist-test-slave-f1t5 krb5kdc[24513](info): commencing operation
krb5kdc: starting...
W20260619 11:13:31.225358 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:13:31.339704 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33289
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:33289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:31.447587 24526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:31.447829 24526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:31.447924 24526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:31.451748 24526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:31.451814 24526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:31.451848 24526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:31.451903 24526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:31.451948 24526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:31.457471 24526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:33289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:31.458422 24526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:31.459367 24526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:31.467705 24531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:31.467700 24532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:31.467872 24534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:31.467753 24526 server_base.cc:1034] running on GCE node
I20260619 11:13:31.468475 24526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:31.469585 24526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:31.470770 24526 hybrid_clock.cc:648] HybridClock initialized: now 1781867611470743 us; error 38 us; skew 500 ppm
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:31.473497 24526 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:31.474395 24526 webserver.cc:460] Webserver started at http://127.7.90.126:42095/ using document root <none> and password file <none>
I20260619 11:13:31.474891 24526 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:31.474954 24526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:31.475117 24526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:31.477034 24526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "8bcffb7cb00b4b3188fbb29114231c8c"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "d3eccbeb2c2ddda520004413f36d08c5"
server_key_iv: "a5f26251fe54f4925fe819cca75d521c"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.477644 24526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "8bcffb7cb00b4b3188fbb29114231c8c"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "d3eccbeb2c2ddda520004413f36d08c5"
server_key_iv: "a5f26251fe54f4925fe819cca75d521c"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.481223 24526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:13:31.483578 24541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.484738 24526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:13:31.484884 24526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "8bcffb7cb00b4b3188fbb29114231c8c"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "d3eccbeb2c2ddda520004413f36d08c5"
server_key_iv: "a5f26251fe54f4925fe819cca75d521c"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.484997 24526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:31.515656 24526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:31.518462 24526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:31.518679 24526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:31.526077 24526 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:33289
I20260619 11:13:31.526090 24593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:33289 every 8 connection(s)
I20260619 11:13:31.527025 24526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:31.529628 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:31.534996 24594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: Bootstrap starting.
I20260619 11:13:31.535211 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24526
I20260619 11:13:31.535321 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:31.535617 7529 external_mini_cluster.cc:1428] Setting key f9c6e1c10607f78f0a2a6e39d94722ef
I20260619 11:13:31.537611 24594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:31.538296 24594 log.cc:824] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:31.540273 24594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: No bootstrap required, opened a new log
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:31.544534 24594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } }
I20260619 11:13:31.544754 24594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:31.544858 24594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcffb7cb00b4b3188fbb29114231c8c, State: Initialized, Role: FOLLOWER
I20260619 11:13:31.545271 24594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } }
I20260619 11:13:31.545413 24594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:31.545477 24594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:31.545612 24594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:31.546418 24594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } }
I20260619 11:13:31.546754 24594 leader_election.cc:302] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bcffb7cb00b4b3188fbb29114231c8c; no voters:
I20260619 11:13:31.547133 24594 leader_election.cc:288] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:31.547187 24599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:31.547355 24599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [term 1 LEADER]: Becoming Leader. State: Replica: 8bcffb7cb00b4b3188fbb29114231c8c, State: Running, Role: LEADER
I20260619 11:13:31.547680 24599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } }
I20260619 11:13:31.548288 24594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:31.548801 24601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } } }
I20260619 11:13:31.548911 24601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:31.549175 24600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bcffb7cb00b4b3188fbb29114231c8c. Latest consensus state: current_term: 1 leader_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcffb7cb00b4b3188fbb29114231c8c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33289 } } }
I20260619 11:13:31.549273 24600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:31.549949 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:31.536864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.7.90.126:33289)
0619 11:13:31.537238 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:31.537254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:31.537294 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:13:31.538000 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:31.538017 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:31.538334 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:31.538664 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:31.539433 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.540443 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.541578 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.542023 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.542151 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:31.544854 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:31.544859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:31.544870 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:31.544894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:31.546841 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:31.547321 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:31.547333 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:31.547337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:31.547488 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:31.547777 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:31.547781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:31.547783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:31.548192 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:31.548403 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:31.548661 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 11:13:31.548940 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
W20260619 11:13:31.553691 24614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:13:31.553766 24614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:13:31.553838 24603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:31.554477 24603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:31.560102 24603 catalog_manager.cc:1347] Generated new cluster ID: f4ad4fb8a4d44109a3085c9efc0a74c3
I20260619 11:13:31.560173 24603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:31.568127 24603 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:31.568977 24603 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:31.574419 24603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: Generated new TSK 0
I20260619 11:13:31.575009 24603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:13:31.604593 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:31.711762 24620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:31.711984 24620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:31.712056 24620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:31.715988 24620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:31.716040 24620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:31.716151 24620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:31.721220 24620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:31.722184 24620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:31.723148 24620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:31.732340 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:31.732363 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:31.732389 24620 server_base.cc:1034] running on GCE node
W20260619 11:13:31.732322 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:31.733034 24620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:31.733639 24620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:31.734835 24620 hybrid_clock.cc:648] HybridClock initialized: now 1781867611734813 us; error 32 us; skew 500 ppm
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:31.737489 24620 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:13:31.738306 24620 webserver.cc:460] Webserver started at http://127.7.90.65:46201/ using document root <none> and password file <none>
I20260619 11:13:31.738850 24620 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:31.738956 24620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:31.739171 24620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:31.741187 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "4948f417a911431eadeef5349b07a6ce"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "66663c592d78a3df8fcb4bed3c35ac0d"
server_key_iv: "c8a411d16d9ec65b6a996245e140916f"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.741777 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "4948f417a911431eadeef5349b07a6ce"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "66663c592d78a3df8fcb4bed3c35ac0d"
server_key_iv: "c8a411d16d9ec65b6a996245e140916f"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.745157 24620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:13:31.747510 24635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.748710 24620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:31.748849 24620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "4948f417a911431eadeef5349b07a6ce"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "66663c592d78a3df8fcb4bed3c35ac0d"
server_key_iv: "c8a411d16d9ec65b6a996245e140916f"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.748963 24620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:31.780190 24620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:31.783010 24620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:31.783156 24620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:31.783644 24620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:31.784379 24620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:31.784425 24620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.784461 24620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:31.784476 24620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.793481 24620 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:44563
I20260619 11:13:31.793584 24748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:44563 every 8 connection(s)
I20260619 11:13:31.794399 24620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:31.800738 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24620
I20260619 11:13:31.800879 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:13:31.801137 7529 external_mini_cluster.cc:1428] Setting key 4c4c1673075289f5a5e161c7161f8627
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:31.807430 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:31.796167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:39669 (local address 127.7.90.126:33289)
0619 11:13:31.796313 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:31.796317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:31.797096 (+ 779us) server_negotiation.cc:407] Connection header received
0619 11:13:31.798001 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:31.798004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:31.798061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:31.798156 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:31.799220 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.799743 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.800825 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.801106 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.801219 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:31.803827 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:31.803834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:31.803836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:31.803864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:31.805197 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:31.805759 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:31.805763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:31.805765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:31.805842 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:31.806207 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:31.806212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:31.806217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:31.806473 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:31.806577 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:31.807053 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 11:13:31.807198 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:13:31.807945 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:31.796511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33289 (local address 127.7.90.65:39669)
0619 11:13:31.796951 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:31.796992 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:31.797802 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:31.798315 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:31.798321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:31.798638 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:31.799073 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:31.799086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.799875 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.799879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:31.800698 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:31.800706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:31.801250 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:31.801254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:31.801419 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:31.802072 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:31.802090 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:31.803663 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:31.805332 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:31.805337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:31.805348 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:31.805662 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:31.805926 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:31.805930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:31.805933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:31.806066 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:31.806573 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:31.806577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:31.806774 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 11:13:31.806940 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 11:13:31.807160 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260619 11:13:31.809067 24749 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33289
I20260619 11:13:31.809280 24749 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:31.809755 24749 heartbeater.cc:511] Master 127.7.90.126:33289 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:13:31.811125 24558 ts_manager.cc:194] Registered new tserver with Master: 4948f417a911431eadeef5349b07a6ce (127.7.90.65:44563)
I20260619 11:13:31.812206 24558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:39669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:13:31.826503 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:31.935814 24754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:31.936046 24754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:31.936085 24754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:31.939888 24754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:31.939936 24754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:31.939989 24754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:31.944891 24754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:31.945784 24754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:31.946669 24754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:31.955737 24760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:31.955737 24759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:31.955814 24754 server_base.cc:1034] running on GCE node
W20260619 11:13:31.955822 24762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:31.956378 24754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:31.956954 24754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:31.958141 24754 hybrid_clock.cc:648] HybridClock initialized: now 1781867611958127 us; error 29 us; skew 500 ppm
Jun 19 11:13:31 dist-test-slave-f1t5 krb5kdc[24513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:31.960673 24754 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:13:31.961548 24754 webserver.cc:460] Webserver started at http://127.7.90.66:40477/ using document root <none> and password file <none>
I20260619 11:13:31.962098 24754 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:31.962188 24754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:31.962401 24754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:31.964270 24754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "a005af6237464075ba4f015bdd06e119"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "2609d500430b9e0f4404f4b47adeee5c"
server_key_iv: "463df562aff36033c7caf7b6967c21f6"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.964808 24754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "a005af6237464075ba4f015bdd06e119"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "2609d500430b9e0f4404f4b47adeee5c"
server_key_iv: "463df562aff36033c7caf7b6967c21f6"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.968286 24754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:13:31.970638 24769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.971827 24754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:13:31.971966 24754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "a005af6237464075ba4f015bdd06e119"
format_stamp: "Formatted at 2026-06-19 11:13:31 on dist-test-slave-f1t5"
server_key: "2609d500430b9e0f4404f4b47adeee5c"
server_key_iv: "463df562aff36033c7caf7b6967c21f6"
server_key_version: "encryptionkey@0"
I20260619 11:13:31.972074 24754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:31.990744 24754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:31.993853 24754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:31.994059 24754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:31.994623 24754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:31.995401 24754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:31.995471 24754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:31.995554 24754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:31.995604 24754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.004240 24754 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:34197
I20260619 11:13:32.004282 24882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:34197 every 8 connection(s)
I20260619 11:13:32.005095 24754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:32.011921 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24754
I20260619 11:13:32.012048 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:32.012337 7529 external_mini_cluster.cc:1428] Setting key 0c23ff2a6921b4256e2ede9e50f4c476
Jun 19 11:13:32 dist-test-slave-f1t5 krb5kdc[24513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867611, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:32.017961 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.006702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:43033 (local address 127.7.90.126:33289)
0619 11:13:32.006866 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.006870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.007670 (+ 800us) server_negotiation.cc:407] Connection header received
0619 11:13:32.008481 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.008484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.008530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.008607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:32.009799 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.010310 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.011360 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.011566 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.011640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.014231 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:32.014234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:32.014236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:32.014257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:32.015952 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.016422 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.016426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.016430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.016521 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.016845 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.016848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.016850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.017034 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:32.017141 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.017667 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.017793 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 11:13:32.018740 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.007089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33289 (local address 127.7.90.66:43033)
0619 11:13:32.007519 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.007560 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.008287 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.008769 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.008776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.009106 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:32.009675 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.009691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.010411 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.010413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.011247 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.011255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.011686 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.011688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.011831 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.012468 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:32.012490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:32.014066 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:32.016072 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:32.016077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:32.016088 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:32.016325 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:32.016618 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:32.016620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:32.016622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:32.016743 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:32.017150 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:32.017158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:32.017385 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.017612 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.017934 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
I20260619 11:13:32.020020 24883 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33289
I20260619 11:13:32.020243 24883 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:32.020695 24883 heartbeater.cc:511] Master 127.7.90.126:33289 requested a full tablet report, sending...
I20260619 11:13:32.021744 24558 ts_manager.cc:194] Registered new tserver with Master: a005af6237464075ba4f015bdd06e119 (127.7.90.66:34197)
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:13:32.022346 24558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:43033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:13:32.038371 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:32.144474 24888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:32.144678 24888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:32.144716 24888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:32.148576 24888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:32.148622 24888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:32.148706 24888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:32.154129 24888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:32.155004 24888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:32.155905 24888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:32.164968 24893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:32.165021 24894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:32.165071 24888 server_base.cc:1034] running on GCE node
W20260619 11:13:32.165022 24896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:32.165621 24888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:32.166204 24888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:32.167383 24888 hybrid_clock.cc:648] HybridClock initialized: now 1781867612167353 us; error 41 us; skew 500 ppm
Jun 19 11:13:32 dist-test-slave-f1t5 krb5kdc[24513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867612, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:32.169942 24888 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:13:32.170727 24888 webserver.cc:460] Webserver started at http://127.7.90.67:38133/ using document root <none> and password file <none>
I20260619 11:13:32.171260 24888 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:32.171346 24888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:32.171558 24888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:32.173463 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "b955d0599b08142096e9b52769ed289e"
server_key_iv: "092cbde06924681bcbb355544c4dccc2"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.174088 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "b955d0599b08142096e9b52769ed289e"
server_key_iv: "092cbde06924681bcbb355544c4dccc2"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.177611 24888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 11:13:32.179782 24903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.180888 24888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:32.181005 24888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "b955d0599b08142096e9b52769ed289e"
server_key_iv: "092cbde06924681bcbb355544c4dccc2"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.181100 24888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:32.202970 24888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:32.205965 24888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:32.206157 24888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:32.206815 24888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:32.207607 24888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:32.207647 24888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.207703 24888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:32.207736 24888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.215687 24888 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:36817
I20260619 11:13:32.215773 25016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:36817 every 8 connection(s)
I20260619 11:13:32.216493 24888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:32.223883 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 24888
I20260619 11:13:32.224004 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:32.224282 7529 external_mini_cluster.cc:1428] Setting key 937ffa73b1223e0abcc39f0d43c702b4
Jun 19 11:13:32 dist-test-slave-f1t5 krb5kdc[24513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867612, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:32.228790 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.218187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:55647 (local address 127.7.90.126:33289)
0619 11:13:32.218315 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.218318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.219051 (+ 733us) server_negotiation.cc:407] Connection header received
0619 11:13:32.219932 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.219935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.219992 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.220096 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:32.221150 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.221717 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.222825 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.223051 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.223161 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.225666 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:32.225669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:32.225672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:32.225723 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:32.227096 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.227558 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.227561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.227562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.227632 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.227925 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.227927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.227928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.228091 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:32.228165 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.228568 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.228664 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:13:32.229611 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.218504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33289 (local address 127.7.90.67:55647)
0619 11:13:32.218919 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.218957 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.219733 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.220262 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.220269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.220592 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:32.221009 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.221021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.221836 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.221844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.222698 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.222707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.223193 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.223197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.223362 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.223908 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:32.223923 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:32.225459 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:32.227193 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:32.227197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:32.227207 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:32.227476 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:32.227706 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:32.227708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:32.227710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:32.227835 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:32.228174 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:32.228178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:32.228381 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.228525 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.228752 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260619 11:13:32.230722 25017 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33289
I20260619 11:13:32.230954 25017 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:32.231385 25017 heartbeater.cc:511] Master 127.7.90.126:33289 requested a full tablet report, sending...
I20260619 11:13:32.232360 24558 ts_manager.cc:194] Registered new tserver with Master: 9565c7bbd0fb45709c4eccaaf2a9c2f8 (127.7.90.67:36817)
I20260619 11:13:32.232920 24558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:55647
I20260619 11:13:32.237803 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:32.246368 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.239566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.7.90.126:33289)
0619 11:13:32.239693 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.239699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.239856 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:13:32.239979 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.239983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.240022 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.240097 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:32.240771 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.241236 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.242170 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.242374 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.242495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.243682 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:32.243686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:32.243688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:32.243716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:32.244762 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.245150 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.245154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.245156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.245228 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:32.245490 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:32.245494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:32.245496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:32.245725 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:32.245856 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.246117 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.246224 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:13:32.248831 24558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:32.251709 24558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:32.264775 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.259373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.7.90.67:36817)
0619 11:13:32.259718 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.259728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.259750 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:13:32.259900 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.260023 (+ 123us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.260212 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.260331 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:32.260806 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.261727 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.263153 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.263986 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.264110 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.264257 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.264483 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.264592 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260619 11:13:32.264806 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.259228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:36817 (local address 127.0.0.1:51826)
0619 11:13:32.259553 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.259600 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.259720 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.260517 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.260523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.260544 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:32.260668 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.260673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.261853 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.261856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.263010 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.263020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.264124 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.264128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.264243 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.264314 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.264426 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.264622 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":70,"thread_start_us":90,"threads_started":1}
I20260619 11:13:32.264987 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.259649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44563 (local address 127.0.0.1:39492)
0619 11:13:32.259877 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.259888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.259967 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.260715 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.260718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.260735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:32.260862 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.260868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.262265 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.262268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.263193 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.263202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.264239 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.264243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.264355 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.264371 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.264470 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.264600 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260619 11:13:32.265446 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.259919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:34197 (local address 127.0.0.1:36904)
0619 11:13:32.260409 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.260421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.260494 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.261368 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.261371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.261387 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:32.261457 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.261462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.262642 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.262646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.263694 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.263703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.265084 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.265090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.265179 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.265195 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.265243 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.265327 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":81,"threads_started":1}
I20260619 11:13:32.265583 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.260025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.7.90.65:44563)
0619 11:13:32.260293 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.260307 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.260328 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:13:32.260399 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.260407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.260573 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.260999 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:32.261154 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.262133 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.263330 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.264111 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.265113 (+ 1002us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.265268 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.265319 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.265407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260619 11:13:32.266014 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.260277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.7.90.66:34197)
0619 11:13:32.260797 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.260808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.260828 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:13:32.260898 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.260905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.261071 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.261204 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:32.261596 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.262486 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.263840 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.264656 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.264759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.265716 (+ 957us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.265763 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.265853 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":114,"threads_started":1}
I20260619 11:13:32.267974 24683 tablet_service.cc:1467] Processing CreateTablet for tablet 2499426673a0489f87418615a852bc92 (DEFAULT_TABLE table=test-table [id=3ad3dc2e279c4096b53d80c7f1226807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:32.268065 24951 tablet_service.cc:1467] Processing CreateTablet for tablet 2499426673a0489f87418615a852bc92 (DEFAULT_TABLE table=test-table [id=3ad3dc2e279c4096b53d80c7f1226807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:32.268441 24817 tablet_service.cc:1467] Processing CreateTablet for tablet 2499426673a0489f87418615a852bc92 (DEFAULT_TABLE table=test-table [id=3ad3dc2e279c4096b53d80c7f1226807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:32.269002 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2499426673a0489f87418615a852bc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:32.269047 24951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2499426673a0489f87418615a852bc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:32.269296 24817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2499426673a0489f87418615a852bc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:32.274998 25034 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Bootstrap starting.
I20260619 11:13:32.276607 25036 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Bootstrap starting.
I20260619 11:13:32.277441 25034 tablet_bootstrap.cc:654] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:32.278478 25034 log.cc:824] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:32.278677 25036 tablet_bootstrap.cc:654] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:32.279122 25035 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Bootstrap starting.
I20260619 11:13:32.279546 25036 log.cc:824] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:32.280483 25034 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: No bootstrap required, opened a new log
I20260619 11:13:32.280673 25034 ts_tablet_manager.cc:1397] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:13:32.281008 25035 tablet_bootstrap.cc:654] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:32.281498 25036 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: No bootstrap required, opened a new log
I20260619 11:13:32.281749 25036 ts_tablet_manager.cc:1397] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 11:13:32.282055 25035 log.cc:824] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:32.283807 25035 tablet_bootstrap.cc:492] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: No bootstrap required, opened a new log
I20260619 11:13:32.284003 25035 ts_tablet_manager.cc:1397] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:13:32.284561 25034 raft_consensus.cc:348] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.284822 25034 raft_consensus.cc:374] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:32.284940 25034 raft_consensus.cc:729] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4948f417a911431eadeef5349b07a6ce, State: Initialized, Role: FOLLOWER
I20260619 11:13:32.285392 25034 consensus_queue.cc:260] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.285471 25036 raft_consensus.cc:348] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.285861 25036 raft_consensus.cc:374] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:32.285952 25036 raft_consensus.cc:729] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a005af6237464075ba4f015bdd06e119, State: Initialized, Role: FOLLOWER
I20260619 11:13:32.286118 24749 heartbeater.cc:503] Master 127.7.90.126:33289 was elected leader, sending a full tablet report...
I20260619 11:13:32.286471 25034 ts_tablet_manager.cc:1428] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 11:13:32.286413 25036 consensus_queue.cc:260] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.287056 24883 heartbeater.cc:503] Master 127.7.90.126:33289 was elected leader, sending a full tablet report...
I20260619 11:13:32.287317 25036 ts_tablet_manager.cc:1428] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 11:13:32.288920 25035 raft_consensus.cc:348] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.289180 25035 raft_consensus.cc:374] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:32.289283 25035 raft_consensus.cc:729] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9565c7bbd0fb45709c4eccaaf2a9c2f8, State: Initialized, Role: FOLLOWER
I20260619 11:13:32.289680 25035 consensus_queue.cc:260] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.290647 25035 ts_tablet_manager.cc:1428] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:13:32.291005 25017 heartbeater.cc:503] Master 127.7.90.126:33289 was elected leader, sending a full tablet report...
W20260619 11:13:32.295435 24750 tablet.cc:2369] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:32.341020 25042 raft_consensus.cc:482] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:32.341275 25042 raft_consensus.cc:504] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.342509 25042 leader_election.cc:288] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a005af6237464075ba4f015bdd06e119 (127.7.90.66:34197), 4948f417a911431eadeef5349b07a6ce (127.7.90.65:44563)
I20260619 11:13:32.346274 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.342671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:34197 (local address 127.7.90.67:59677)
0619 11:13:32.342792 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.342819 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.342910 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.343162 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.343164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.343181 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:32.343356 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.343363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.344188 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.344191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.345047 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.345055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.345839 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.345954 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.345978 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.346052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.346126 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260619 11:13:32.346290 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.342790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:59677 (local address 127.7.90.66:34197)
0619 11:13:32.342932 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.342937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.342953 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:13:32.343010 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.343013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.343060 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.343145 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:32.343464 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.344086 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345162 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.345710 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345791 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.345845 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.346080 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.346153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:13:32.346441 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.342951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:35005 (local address 127.7.90.65:44563)
0619 11:13:32.343073 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.343076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.343235 (+ 159us) server_negotiation.cc:407] Connection header received
0619 11:13:32.343362 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.343364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.343431 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.343509 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:32.343756 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.344292 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345357 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.345872 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345970 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.346031 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.346233 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.346321 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:13:32.346478 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.342745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44563 (local address 127.7.90.67:35005)
0619 11:13:32.343142 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.343156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.343250 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.343533 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.343536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.343554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:32.343654 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.343660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.344417 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.344420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.345242 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.345251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.345983 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.345986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.346105 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.346128 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.346204 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.346267 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":105,"threads_started":1}
I20260619 11:13:32.346980 24703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2499426673a0489f87418615a852bc92" candidate_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4948f417a911431eadeef5349b07a6ce" is_pre_election: true
I20260619 11:13:32.347111 24837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2499426673a0489f87418615a852bc92" candidate_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a005af6237464075ba4f015bdd06e119" is_pre_election: true
I20260619 11:13:32.347281 24703 raft_consensus.cc:2454] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565c7bbd0fb45709c4eccaaf2a9c2f8 in term 0.
I20260619 11:13:32.347441 24837 raft_consensus.cc:2454] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565c7bbd0fb45709c4eccaaf2a9c2f8 in term 0.
I20260619 11:13:32.347807 24907 leader_election.cc:302] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4948f417a911431eadeef5349b07a6ce, 9565c7bbd0fb45709c4eccaaf2a9c2f8; no voters:
I20260619 11:13:32.348094 25042 raft_consensus.cc:2781] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:32.348178 25042 raft_consensus.cc:482] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:32.348212 25042 raft_consensus.cc:3037] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:32.349448 25042 raft_consensus.cc:504] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.349845 25042 leader_election.cc:288] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [CANDIDATE]: Term 1 election: Requested vote from peers a005af6237464075ba4f015bdd06e119 (127.7.90.66:34197), 4948f417a911431eadeef5349b07a6ce (127.7.90.65:44563)
I20260619 11:13:32.350162 24837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2499426673a0489f87418615a852bc92" candidate_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a005af6237464075ba4f015bdd06e119"
I20260619 11:13:32.350217 24703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2499426673a0489f87418615a852bc92" candidate_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4948f417a911431eadeef5349b07a6ce"
I20260619 11:13:32.350287 24837 raft_consensus.cc:3037] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:32.350353 24703 raft_consensus.cc:3037] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:32.351343 24837 raft_consensus.cc:2454] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565c7bbd0fb45709c4eccaaf2a9c2f8 in term 1.
I20260619 11:13:32.351387 24703 raft_consensus.cc:2454] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565c7bbd0fb45709c4eccaaf2a9c2f8 in term 1.
I20260619 11:13:32.351744 24907 leader_election.cc:302] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4948f417a911431eadeef5349b07a6ce, 9565c7bbd0fb45709c4eccaaf2a9c2f8; no voters:
I20260619 11:13:32.351958 25042 raft_consensus.cc:2781] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:32.352171 25042 raft_consensus.cc:686] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9565c7bbd0fb45709c4eccaaf2a9c2f8, State: Running, Role: LEADER
I20260619 11:13:32.352514 25042 consensus_queue.cc:237] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } }
I20260619 11:13:32.355836 24558 catalog_manager.cc:5495] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9565c7bbd0fb45709c4eccaaf2a9c2f8 (127.7.90.67). New cstate: current_term: 1 leader_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565c7bbd0fb45709c4eccaaf2a9c2f8" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 36817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 } health_report { overall_health: UNKNOWN } } }
I20260619 11:13:32.387401 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.383975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.7.90.67:36817)
0619 11:13:32.384133 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.384140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.384158 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:13:32.384240 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.384242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.384292 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.384397 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:32.384663 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.385143 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.386123 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.386321 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.386402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.386634 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:32.387017 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:32.387101 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.387157 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.387203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:13:32.393154 24703 raft_consensus.cc:1261] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce [term 1 FOLLOWER]: Refusing update from remote peer 9565c7bbd0fb45709c4eccaaf2a9c2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:32.393153 24837 raft_consensus.cc:1261] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 1 FOLLOWER]: Refusing update from remote peer 9565c7bbd0fb45709c4eccaaf2a9c2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:32.393934 25042 consensus_queue.cc:1035] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4948f417a911431eadeef5349b07a6ce" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:32.394169 25044 consensus_queue.cc:1035] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a005af6237464075ba4f015bdd06e119" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 34197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:32.400696 25047 mvcc.cc:205] Tried to move back new op lower bound from 7298529740355268608 to 7298529740198490112. Current Snapshot: MvccSnapshot[applied={T|T < 7298529740355268608}]
I20260619 11:13:32.401719 25049 mvcc.cc:205] Tried to move back new op lower bound from 7298529740355268608 to 7298529740198490112. Current Snapshot: MvccSnapshot[applied={T|T < 7298529740355268608}]
I20260619 11:13:32.410099 24557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:32.410225 24557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:32.412477 24557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bcffb7cb00b4b3188fbb29114231c8c: Sending DeleteTablet for 3 replicas of tablet 2499426673a0489f87418615a852bc92
I20260619 11:13:32.413220 24683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2499426673a0489f87418615a852bc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:32 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:39492
I20260619 11:13:32.413255 24817 tablet_service.cc:1514] Processing DeleteTablet for tablet 2499426673a0489f87418615a852bc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:32 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:36904
I20260619 11:13:32.413355 24951 tablet_service.cc:1514] Processing DeleteTablet for tablet 2499426673a0489f87418615a852bc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:32 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:51826
I20260619 11:13:32.413753 25058 tablet_replica.cc:330] T 2499426673a0489f87418615a852bc92 P 4948f417a911431eadeef5349b07a6ce: stopping tablet replica
I20260619 11:13:32.413911 25057 tablet_replica.cc:330] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: stopping tablet replica
I20260619 11:13:32.413969 25059 tablet_replica.cc:330] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: stopping tablet replica
I20260619 11:13:32.414219 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24620
I20260619 11:13:32.415447 25057 raft_consensus.cc:2229] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:32.415661 25059 raft_consensus.cc:2229] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:13:32.415720 25057 raft_consensus.cc:2258] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:32.416018 25059 raft_consensus.cc:2258] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:32.417394 25059 ts_tablet_manager.cc:1905] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:32.417995 25057 ts_tablet_manager.cc:1905] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:32.420619 25057 ts_tablet_manager.cc:1918] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:32.420771 25057 log.cc:1196] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/2499426673a0489f87418615a852bc92
I20260619 11:13:32.420845 25059 ts_tablet_manager.cc:1918] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:32.420998 25059 log.cc:1196] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/2499426673a0489f87418615a852bc92
I20260619 11:13:32.421351 25057 ts_tablet_manager.cc:1939] T 2499426673a0489f87418615a852bc92 P a005af6237464075ba4f015bdd06e119: Deleting consensus metadata
I20260619 11:13:32.421353 25059 ts_tablet_manager.cc:1939] T 2499426673a0489f87418615a852bc92 P 9565c7bbd0fb45709c4eccaaf2a9c2f8: Deleting consensus metadata
I20260619 11:13:32.422516 24544 catalog_manager.cc:4841] TS 9565c7bbd0fb45709c4eccaaf2a9c2f8 (127.7.90.67:36817): tablet 2499426673a0489f87418615a852bc92 (table test-table [id=3ad3dc2e279c4096b53d80c7f1226807]) successfully deleted
I20260619 11:13:32.422701 24544 catalog_manager.cc:4841] TS a005af6237464075ba4f015bdd06e119 (127.7.90.66:34197): tablet 2499426673a0489f87418615a852bc92 (table test-table [id=3ad3dc2e279c4096b53d80c7f1226807]) successfully deleted
W20260619 11:13:32.423084 24545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:44563 (error 108)
I20260619 11:13:32.423313 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24754
I20260619 11:13:32.424352 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.424036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:44563 (local address 127.0.0.1:39504)
0619 11:13:32.424213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.424277 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 11:13:32.424536 24545 catalog_manager.cc:4584] TS 4948f417a911431eadeef5349b07a6ce (127.7.90.65:44563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2499426673a0489f87418615a852bc92: Network error: Client connection negotiation failed: client connection to 127.7.90.65:44563: connect: Connection refused (error 111)
I20260619 11:13:32.429946 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24888
I20260619 11:13:32.435192 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 24526
2026-06-19T11:13:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2244 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T11:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:32Z Disabled control of system clock
I20260619 11:13:32.464393 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33137
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:33137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:32.571069 25064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:32.571326 25064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:32.571362 25064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:32.575304 25064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:32.575353 25064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:32.575368 25064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:32.575384 25064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:32.575399 25064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:32.579950 25064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:33137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:33137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:32.580909 25064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:32.582211 25064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:32.590183 25069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:32.590175 25072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:32.590325 25064 server_base.cc:1034] running on GCE node
W20260619 11:13:32.590477 25070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:32.590977 25064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:32.592001 25064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:32.593163 25064 hybrid_clock.cc:648] HybridClock initialized: now 1781867612593129 us; error 44 us; skew 500 ppm
I20260619 11:13:32.594945 25064 webserver.cc:460] Webserver started at http://127.7.90.126:40581/ using document root <none> and password file <none>
I20260619 11:13:32.595496 25064 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:32.595556 25064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:32.595799 25064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:32.597812 25064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "2c1c5e2616794eb9a1582b4fc6339009"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "d89c470d6aa8909930bb0559620033f5"
server_key_iv: "f01bae2b854622f77aa794716e456028"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.598356 25064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "2c1c5e2616794eb9a1582b4fc6339009"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "d89c470d6aa8909930bb0559620033f5"
server_key_iv: "f01bae2b854622f77aa794716e456028"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.601728 25064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:13:32.603869 25078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.604940 25064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:32.605069 25064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2c1c5e2616794eb9a1582b4fc6339009"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "d89c470d6aa8909930bb0559620033f5"
server_key_iv: "f01bae2b854622f77aa794716e456028"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.605170 25064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:32.625969 25064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:32.626542 25064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:32.626755 25064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:32.633190 25064 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:33137
I20260619 11:13:32.633224 25130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:33137 every 8 connection(s)
I20260619 11:13:32.634056 25064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:32.636994 25131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:32.639616 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25064
I20260619 11:13:32.639743 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:32.640053 7529 external_mini_cluster.cc:1428] Setting key f2b66d274082bab31a912f73482a19df
I20260619 11:13:32.642890 25131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: Bootstrap starting.
I20260619 11:13:32.644929 25131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:32.645686 25131 log.cc:824] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:32.647446 25131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: No bootstrap required, opened a new log
I20260619 11:13:32.649078 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.641418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.7.90.126:33137)
0619 11:13:32.642047 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.642060 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.642104 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:13:32.642726 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.642741 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.643155 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.643474 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:32.643895 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.644811 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.645799 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.646438 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.646631 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.647485 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:32.647491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:32.647507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:32.647911 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:32.647963 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:32.647983 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:32.648087 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.648198 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.648456 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260619 11:13:32.651171 25131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } }
I20260619 11:13:32.651391 25131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:32.651465 25131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1c5e2616794eb9a1582b4fc6339009, State: Initialized, Role: FOLLOWER
I20260619 11:13:32.651887 25131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } }
I20260619 11:13:32.652025 25131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:32.652107 25131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:32.652202 25131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:32.652987 25131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } }
I20260619 11:13:32.653306 25131 leader_election.cc:302] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c1c5e2616794eb9a1582b4fc6339009; no voters:
I20260619 11:13:32.653637 25131 leader_election.cc:288] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:32.653739 25136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:32.653992 25136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [term 1 LEADER]: Becoming Leader. State: Replica: 2c1c5e2616794eb9a1582b4fc6339009, State: Running, Role: LEADER
I20260619 11:13:32.654328 25136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } }
I20260619 11:13:32.654709 25131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:32.656013 25138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c1c5e2616794eb9a1582b4fc6339009. Latest consensus state: current_term: 1 leader_uuid: "2c1c5e2616794eb9a1582b4fc6339009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } } }
I20260619 11:13:32.656400 25138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:32.656286 25137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c1c5e2616794eb9a1582b4fc6339009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1c5e2616794eb9a1582b4fc6339009" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 33137 } } }
I20260619 11:13:32.656774 25137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009 [sys.catalog]: This master's current role is: LEADER
W20260619 11:13:32.659108 25151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:13:32.659168 25151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:13:32.659246 25150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:32.659917 25150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:32.664878 25150 catalog_manager.cc:1347] Generated new cluster ID: dc0d43ea81b54a50b6bd92913fb67294
I20260619 11:13:32.664938 25150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:32.677417 25150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:32.678571 25150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:32.683827 25150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: Generated new TSK 0
I20260619 11:13:32.684629 25150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:32.696761 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--builtin_ntp_servers=127.7.90.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:32.805781 25155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:32.806008 25155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:32.806069 25155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:32.809804 25155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:32.809844 25155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:32.809935 25155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:32.814584 25155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:32.815544 25155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:32.816861 25155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:32.825903 25161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:32.825915 25163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:32.826113 25160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:32.826370 25155 server_base.cc:1034] running on GCE node
I20260619 11:13:32.826741 25155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:32.827347 25155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:32.828527 25155 hybrid_clock.cc:648] HybridClock initialized: now 1781867612828520 us; error 29 us; skew 500 ppm
I20260619 11:13:32.830241 25155 webserver.cc:460] Webserver started at http://127.7.90.65:33335/ using document root <none> and password file <none>
I20260619 11:13:32.830783 25155 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:32.830881 25155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:32.831115 25155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:32.833114 25155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "0a5887eb0232451c926e095c3e7065de"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "830b50feff71c41612e86e7da2d52ecb"
server_key_iv: "9ea6f0b1037c33f005dba37f69e0e084"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.833724 25155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "0a5887eb0232451c926e095c3e7065de"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "830b50feff71c41612e86e7da2d52ecb"
server_key_iv: "9ea6f0b1037c33f005dba37f69e0e084"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.837267 25155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:13:32.839650 25169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.840791 25155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:13:32.840951 25155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "0a5887eb0232451c926e095c3e7065de"
format_stamp: "Formatted at 2026-06-19 11:13:32 on dist-test-slave-f1t5"
server_key: "830b50feff71c41612e86e7da2d52ecb"
server_key_iv: "9ea6f0b1037c33f005dba37f69e0e084"
server_key_version: "encryptionkey@0"
I20260619 11:13:32.841058 25155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:32.857620 25155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:32.858254 25155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:32.858443 25155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:32.858994 25155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:32.859880 25155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:32.859956 25155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.860026 25155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:32.860069 25155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:32.868391 25155 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:33997
I20260619 11:13:32.868435 25282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:33997 every 8 connection(s)
I20260619 11:13:32.869251 25155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:32.872743 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25155
I20260619 11:13:32.872867 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:13:32.873252 7529 external_mini_cluster.cc:1428] Setting key a9217ad4d55bee3c38c2445788ff04e1
I20260619 11:13:32.875227 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--builtin_ntp_servers=127.7.90.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:32.879321 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.870941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:45561 (local address 127.7.90.126:33137)
0619 11:13:32.871098 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:32.871103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:32.872099 (+ 996us) server_negotiation.cc:407] Connection header received
0619 11:13:32.873840 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:32.873844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:32.873911 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:32.874015 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:32.874948 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.875557 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.877071 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.877445 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.877589 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.878742 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:32.878748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:32.878754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:32.878771 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:32.878796 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:32.878808 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:32.878886 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:32.879058 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 11:13:32.879125 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:13:32.880013 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:32.871337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33137 (local address 127.7.90.65:45561)
0619 11:13:32.871959 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:32.871998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:32.873607 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:32.874185 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:32.874192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:32.874272 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:32.874804 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.874822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.875681 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.875688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.876955 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:32.876965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:32.877614 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:32.877617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:32.877751 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:32.878438 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:32.878440 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:32.878490 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:32.878575 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:32.878901 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:32.878911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:32.878917 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:13:32.879024 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:13:32.879222 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260619 11:13:32.881268 25283 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33137
I20260619 11:13:32.881486 25283 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:32.881959 25283 heartbeater.cc:511] Master 127.7.90.126:33137 requested a full tablet report, sending...
I20260619 11:13:32.883335 25095 ts_manager.cc:194] Registered new tserver with Master: 0a5887eb0232451c926e095c3e7065de (127.7.90.65:33997)
I20260619 11:13:32.884447 25095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.65:45561
W20260619 11:13:32.980499 25286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:32.980729 25286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:32.980765 25286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:32.984719 25286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:32.984761 25286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:32.984813 25286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:32.989271 25286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:32.990209 25286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:32.991437 25286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:33.000263 25294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:33.000286 25292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:33.000263 25291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:33.000790 25286 server_base.cc:1034] running on GCE node
I20260619 11:13:33.001134 25286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:33.001718 25286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:33.002873 25286 hybrid_clock.cc:648] HybridClock initialized: now 1781867613002858 us; error 27 us; skew 500 ppm
I20260619 11:13:33.004583 25286 webserver.cc:460] Webserver started at http://127.7.90.66:44397/ using document root <none> and password file <none>
I20260619 11:13:33.005074 25286 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:33.005131 25286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:33.005281 25286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:33.007222 25286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e0d80b604a93cdc08aabe231c740c375"
server_key_iv: "dbbb633a4351fcbdc2c434ae8f3c37ff"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.007720 25286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e0d80b604a93cdc08aabe231c740c375"
server_key_iv: "dbbb633a4351fcbdc2c434ae8f3c37ff"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.011204 25286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:33.013355 25300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.014541 25286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:33.014626 25286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e0d80b604a93cdc08aabe231c740c375"
server_key_iv: "dbbb633a4351fcbdc2c434ae8f3c37ff"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.014717 25286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:33.038190 25286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:33.038722 25286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:33.038857 25286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:33.039317 25286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:33.040112 25286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:33.040154 25286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.040190 25286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:33.040211 25286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.049026 25286 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:35253
I20260619 11:13:33.049053 25413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:35253 every 8 connection(s)
I20260619 11:13:33.049847 25286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:33.051319 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25286
I20260619 11:13:33.051450 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:33.051767 7529 external_mini_cluster.cc:1428] Setting key caf2214a60b9e7eaa081c81bed6ae95f
I20260619 11:13:33.053937 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--builtin_ntp_servers=127.7.90.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:33.059369 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.051537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:44831 (local address 127.7.90.126:33137)
0619 11:13:33.051703 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.051708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.052774 (+ 1066us) server_negotiation.cc:407] Connection header received
0619 11:13:33.054597 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.054599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.054642 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.054702 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:33.055811 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.056362 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.057465 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.057747 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.057868 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.058894 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:33.058898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:33.058901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:33.058912 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:33.058936 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:33.058949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:33.059027 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.059176 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.059233 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:13:33.060124 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.051957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33137 (local address 127.7.90.66:44831)
0619 11:13:33.052646 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.052683 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.054372 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.054863 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.054873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.054979 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:33.055668 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.055684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.056489 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.056492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.057342 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.057351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.057898 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.057902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.058063 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.058637 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:33.058639 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:33.058674 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:33.058731 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:33.059028 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:33.059040 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:33.059046 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.059175 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.059433 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
I20260619 11:13:33.061210 25414 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33137
I20260619 11:13:33.061462 25414 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:33.062084 25414 heartbeater.cc:511] Master 127.7.90.126:33137 requested a full tablet report, sending...
I20260619 11:13:33.063143 25095 ts_manager.cc:194] Registered new tserver with Master: efaffc5e6fbf4f988c0a1109a25fe7a7 (127.7.90.66:35253)
I20260619 11:13:33.063684 25095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.66:44831
W20260619 11:13:33.163103 25417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:33.163332 25417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:33.163368 25417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:33.167325 25417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:33.167371 25417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:33.167423 25417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:33.172092 25417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:33.173128 25417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:33.174533 25417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:33.183600 25423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:33.183730 25422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:33.183600 25425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:33.183755 25417 server_base.cc:1034] running on GCE node
I20260619 11:13:33.184322 25417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:33.184930 25417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:33.186110 25417 hybrid_clock.cc:648] HybridClock initialized: now 1781867613186075 us; error 51 us; skew 500 ppm
I20260619 11:13:33.187909 25417 webserver.cc:460] Webserver started at http://127.7.90.67:41393/ using document root <none> and password file <none>
I20260619 11:13:33.188479 25417 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:33.188570 25417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:33.188791 25417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:33.190816 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "d2263960078544e1a009dca61ae4dcc3"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e60df87178662fb411b64fed7efcd820"
server_key_iv: "03e8c064def47626e65c2c88c30343fb"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.191359 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "d2263960078544e1a009dca61ae4dcc3"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e60df87178662fb411b64fed7efcd820"
server_key_iv: "03e8c064def47626e65c2c88c30343fb"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.194885 25417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:33.197319 25431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.198496 25417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:33.198626 25417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "d2263960078544e1a009dca61ae4dcc3"
format_stamp: "Formatted at 2026-06-19 11:13:33 on dist-test-slave-f1t5"
server_key: "e60df87178662fb411b64fed7efcd820"
server_key_iv: "03e8c064def47626e65c2c88c30343fb"
server_key_version: "encryptionkey@0"
I20260619 11:13:33.198729 25417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:33.219508 25417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:33.220112 25417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:33.220297 25417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:33.220830 25417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:33.221767 25417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:33.221843 25417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.221935 25417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:33.221987 25417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:33.230509 25417 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44417
I20260619 11:13:33.230532 25544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44417 every 8 connection(s)
I20260619 11:13:33.231331 25417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:33.240139 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.233019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:53609 (local address 127.7.90.126:33137)
0619 11:13:33.233153 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.233156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.234168 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 11:13:33.235518 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.235522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.235579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.235688 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:33.236530 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.237067 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.238340 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.238572 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.238682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.239649 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:33.239653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:33.239655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:33.239667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:33.239693 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:33.239701 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:33.239774 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.239938 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.239996 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:33.240433 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25417
I20260619 11:13:33.240535 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:33.240808 7529 external_mini_cluster.cc:1428] Setting key cc27d25b524c059e3b9c65c754d6f20a
I20260619 11:13:33.241051 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.233365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:33137 (local address 127.7.90.67:53609)
0619 11:13:33.234020 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.234058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.235313 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.235850 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.235856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.235935 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:33.236382 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.236394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.237191 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.237195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.238202 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.238213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.238716 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.238720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.238884 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.239404 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:33.239405 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:33.239441 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:33.239498 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:33.239788 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:33.239797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:33.239804 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.239933 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.240211 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":173,"threads_started":1}
I20260619 11:13:33.242202 25545 heartbeater.cc:347] Connected to a master server at 127.7.90.126:33137
I20260619 11:13:33.242431 25545 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:33.242973 25545 heartbeater.cc:511] Master 127.7.90.126:33137 requested a full tablet report, sending...
I20260619 11:13:33.243830 25094 ts_manager.cc:194] Registered new tserver with Master: d2263960078544e1a009dca61ae4dcc3 (127.7.90.67:44417)
I20260619 11:13:33.244453 25094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.67:53609
I20260619 11:13:33.254361 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:33.260084 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.256369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.7.90.126:33137)
0619 11:13:33.256467 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.256473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.256636 (+ 163us) server_negotiation.cc:407] Connection header received
0619 11:13:33.256792 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.256795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.256836 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.256924 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:33.257194 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.257708 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.258543 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.258808 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.258915 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.259635 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:33.259638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:33.259641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:33.259653 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:33.259671 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:33.259687 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:33.259763 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.259904 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.259955 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:13:33.262828 25094 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:33.265717 25094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:33.279407 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.273631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:33997 (local address 127.0.0.1:51832)
0619 11:13:33.274589 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.274637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.274771 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.275336 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.275341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.275363 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:33.275489 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.275495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.276613 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.276618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.277984 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.277993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.278847 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.278850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.278954 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.278991 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.279118 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.279231 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260619 11:13:33.279431 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.273743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.7.90.65:33997)
0619 11:13:33.274601 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.274612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.274716 (+ 104us) server_negotiation.cc:407] Connection header received
0619 11:13:33.274954 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.274963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.275176 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.275344 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:33.275610 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.276507 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.278101 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.278720 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.278833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.279011 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.279147 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.279248 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":119,"threads_started":1}
I20260619 11:13:33.280113 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:35253 (local address 127.0.0.1:38048)
0619 11:13:33.274914 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.274928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.275075 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.275558 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.275563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.275581 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:33.275677 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.275682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.277053 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.277056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.278463 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.278474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.279668 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.279672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.279790 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.279816 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.279888 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.279950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":112,"thread_start_us":94,"threads_started":1}
I20260619 11:13:33.280417 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.274739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.7.90.67:44417)
0619 11:13:33.275066 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.275077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.275451 (+ 374us) server_negotiation.cc:407] Connection header received
0619 11:13:33.275534 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.275541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.275718 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.275868 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:33.275997 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.277065 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.278593 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.279367 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.279474 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.279647 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.280185 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.280278 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260619 11:13:33.280125 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.274365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44417 (local address 127.0.0.1:45574)
0619 11:13:33.275237 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.275250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.275348 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.275836 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.275839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.275856 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:33.275925 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.275931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.277174 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.277177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.278468 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.278474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.279726 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.279729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.279792 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.279817 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.279907 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.279977 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":79,"thread_start_us":88,"threads_started":1}
I20260619 11:13:33.280951 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.274088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.7.90.66:35253)
0619 11:13:33.274400 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.274412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.275012 (+ 600us) server_negotiation.cc:407] Connection header received
0619 11:13:33.275242 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.275250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.275438 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.275585 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:33.275784 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.276920 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.278612 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.279317 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.279411 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.279611 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.280384 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.280712 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260619 11:13:33.282908 25217 tablet_service.cc:1467] Processing CreateTablet for tablet ffe3678c721d4f17abdde3154faf5b04 (DEFAULT_TABLE table=test-table [id=96e26bee6a744ca5ae73870b7381b618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:33.283246 25479 tablet_service.cc:1467] Processing CreateTablet for tablet ffe3678c721d4f17abdde3154faf5b04 (DEFAULT_TABLE table=test-table [id=96e26bee6a744ca5ae73870b7381b618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:33.283910 25217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe3678c721d4f17abdde3154faf5b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:33.284235 25479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe3678c721d4f17abdde3154faf5b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:33.284163 25348 tablet_service.cc:1467] Processing CreateTablet for tablet ffe3678c721d4f17abdde3154faf5b04 (DEFAULT_TABLE table=test-table [id=96e26bee6a744ca5ae73870b7381b618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:33.285046 25348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe3678c721d4f17abdde3154faf5b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:33.289875 25562 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Bootstrap starting.
I20260619 11:13:33.291165 25563 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Bootstrap starting.
I20260619 11:13:33.292157 25562 tablet_bootstrap.cc:654] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:33.292188 25564 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Bootstrap starting.
I20260619 11:13:33.293043 25562 log.cc:824] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:33.293357 25563 tablet_bootstrap.cc:654] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:33.294005 25564 tablet_bootstrap.cc:654] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:33.294214 25563 log.cc:824] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:33.294842 25564 log.cc:824] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:33.295413 25562 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: No bootstrap required, opened a new log
I20260619 11:13:33.295640 25562 ts_tablet_manager.cc:1397] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:13:33.296058 25563 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: No bootstrap required, opened a new log
I20260619 11:13:33.296247 25563 ts_tablet_manager.cc:1397] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:13:33.296766 25564 tablet_bootstrap.cc:492] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: No bootstrap required, opened a new log
I20260619 11:13:33.296947 25564 ts_tablet_manager.cc:1397] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:13:33.298907 25562 raft_consensus.cc:348] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.299124 25562 raft_consensus.cc:374] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:33.299191 25562 raft_consensus.cc:729] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5887eb0232451c926e095c3e7065de, State: Initialized, Role: FOLLOWER
I20260619 11:13:33.299623 25562 consensus_queue.cc:260] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.300216 25283 heartbeater.cc:503] Master 127.7.90.126:33137 was elected leader, sending a full tablet report...
I20260619 11:13:33.300253 25563 raft_consensus.cc:348] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.300470 25563 raft_consensus.cc:374] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:33.300513 25563 raft_consensus.cc:729] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2263960078544e1a009dca61ae4dcc3, State: Initialized, Role: FOLLOWER
I20260619 11:13:33.300536 25562 ts_tablet_manager.cc:1428] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:13:33.300868 25564 raft_consensus.cc:348] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.301093 25564 raft_consensus.cc:374] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:33.301328 25564 raft_consensus.cc:729] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaffc5e6fbf4f988c0a1109a25fe7a7, State: Initialized, Role: FOLLOWER
I20260619 11:13:33.301340 25563 consensus_queue.cc:260] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.301811 25564 consensus_queue.cc:260] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.302335 25563 ts_tablet_manager.cc:1428] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:13:33.302676 25564 ts_tablet_manager.cc:1428] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:13:33.302673 25545 heartbeater.cc:503] Master 127.7.90.126:33137 was elected leader, sending a full tablet report...
I20260619 11:13:33.303081 25414 heartbeater.cc:503] Master 127.7.90.126:33137 was elected leader, sending a full tablet report...
I20260619 11:13:33.351119 25568 raft_consensus.cc:482] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:33.351320 25568 raft_consensus.cc:504] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.352421 25568 leader_election.cc:288] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2263960078544e1a009dca61ae4dcc3 (127.7.90.67:44417), efaffc5e6fbf4f988c0a1109a25fe7a7 (127.7.90.66:35253)
I20260619 11:13:33.356946 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.352644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44417 (local address 127.7.90.65:40001)
0619 11:13:33.352805 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.352822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.352923 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.353189 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.353193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.353218 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:33.353355 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.353370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.354434 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.354436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.355738 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.355749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.356458 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.356462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.356593 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.356634 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.356705 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.356783 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 11:13:33.356952 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.352774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:40001 (local address 127.7.90.67:44417)
0619 11:13:33.352924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.352929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.352947 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:13:33.353014 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.353020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.353072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.353164 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:33.353473 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.354328 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.355941 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.356347 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.356431 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.356478 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.356721 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.356798 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:13:33.357108 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.352783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:46003 (local address 127.7.90.66:35253)
0619 11:13:33.352970 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.352977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.353287 (+ 310us) server_negotiation.cc:407] Connection header received
0619 11:13:33.353354 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.353356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.353397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.353492 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:33.353790 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.354498 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.355984 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.356567 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.356667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.356732 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.356918 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.356992 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 11:13:33.357414 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.352649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:35253 (local address 127.7.90.65:46003)
0619 11:13:33.353177 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.353189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:33.353268 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:33.353566 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:33.353569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:33.353594 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:33.353683 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.353687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.354587 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.354590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.355815 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:33.355824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.356683 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.356687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:33.356791 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.356814 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:13:33.356891 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:13:33.356956 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":217,"thread_start_us":120,"threads_started":1}
I20260619 11:13:33.357635 25499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe3678c721d4f17abdde3154faf5b04" candidate_uuid: "0a5887eb0232451c926e095c3e7065de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2263960078544e1a009dca61ae4dcc3" is_pre_election: true
I20260619 11:13:33.357967 25499 raft_consensus.cc:2454] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5887eb0232451c926e095c3e7065de in term 0.
I20260619 11:13:33.358111 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe3678c721d4f17abdde3154faf5b04" candidate_uuid: "0a5887eb0232451c926e095c3e7065de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" is_pre_election: true
I20260619 11:13:33.358383 25368 raft_consensus.cc:2454] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5887eb0232451c926e095c3e7065de in term 0.
I20260619 11:13:33.358486 25173 leader_election.cc:302] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5887eb0232451c926e095c3e7065de, d2263960078544e1a009dca61ae4dcc3; no voters:
I20260619 11:13:33.358785 25568 raft_consensus.cc:2781] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:33.358866 25568 raft_consensus.cc:482] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:33.358917 25568 raft_consensus.cc:3037] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:33.359975 25568 raft_consensus.cc:504] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.360297 25568 leader_election.cc:288] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [CANDIDATE]: Term 1 election: Requested vote from peers d2263960078544e1a009dca61ae4dcc3 (127.7.90.67:44417), efaffc5e6fbf4f988c0a1109a25fe7a7 (127.7.90.66:35253)
I20260619 11:13:33.360625 25499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe3678c721d4f17abdde3154faf5b04" candidate_uuid: "0a5887eb0232451c926e095c3e7065de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2263960078544e1a009dca61ae4dcc3"
I20260619 11:13:33.360634 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe3678c721d4f17abdde3154faf5b04" candidate_uuid: "0a5887eb0232451c926e095c3e7065de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7"
I20260619 11:13:33.360745 25499 raft_consensus.cc:3037] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:33.360745 25368 raft_consensus.cc:3037] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:33.361987 25368 raft_consensus.cc:2454] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5887eb0232451c926e095c3e7065de in term 1.
I20260619 11:13:33.361987 25499 raft_consensus.cc:2454] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5887eb0232451c926e095c3e7065de in term 1.
I20260619 11:13:33.362278 25173 leader_election.cc:302] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5887eb0232451c926e095c3e7065de, d2263960078544e1a009dca61ae4dcc3; no voters:
I20260619 11:13:33.362457 25568 raft_consensus.cc:2781] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:33.362673 25568 raft_consensus.cc:686] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 1 LEADER]: Becoming Leader. State: Replica: 0a5887eb0232451c926e095c3e7065de, State: Running, Role: LEADER
I20260619 11:13:33.363037 25568 consensus_queue.cc:237] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } }
I20260619 11:13:33.366011 25095 catalog_manager.cc:5495] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a5887eb0232451c926e095c3e7065de (127.7.90.65). New cstate: current_term: 1 leader_uuid: "0a5887eb0232451c926e095c3e7065de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a5887eb0232451c926e095c3e7065de" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 } health_report { overall_health: UNKNOWN } } }
W20260619 11:13:33.370301 25284 tablet.cc:2369] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:33.399835 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.396390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.7.90.65:33997)
0619 11:13:33.396532 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:33.396538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:33.396557 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:13:33.396642 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:33.396644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:33.396701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:33.396833 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:33.397055 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.397577 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.398429 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:33.398645 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:33.398758 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:33.399054 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:33.399434 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:33.399568 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:33.399629 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 11:13:33.399677 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:13:33.405666 25499 raft_consensus.cc:1261] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 1 FOLLOWER]: Refusing update from remote peer 0a5887eb0232451c926e095c3e7065de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:33.405687 25368 raft_consensus.cc:1261] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 1 FOLLOWER]: Refusing update from remote peer 0a5887eb0232451c926e095c3e7065de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:33.406432 25572 consensus_queue.cc:1035] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2263960078544e1a009dca61ae4dcc3" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:33.406613 25568 consensus_queue.cc:1035] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [LEADER]: Connected to new peer: Peer: permanent_uuid: "efaffc5e6fbf4f988c0a1109a25fe7a7" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:33.412240 25575 mvcc.cc:205] Tried to move back new op lower bound from 7298529744502566912 to 7298529744337616896. Current Snapshot: MvccSnapshot[applied={T|T < 7298529744502566912}]
I20260619 11:13:33.413884 25577 mvcc.cc:205] Tried to move back new op lower bound from 7298529744502566912 to 7298529744337616896. Current Snapshot: MvccSnapshot[applied={T|T < 7298529744502566912}]
I20260619 11:13:33.417428 25576 mvcc.cc:205] Tried to move back new op lower bound from 7298529744502566912 to 7298529744337616896. Current Snapshot: MvccSnapshot[applied={T|T < 7298529744502566912}]
I20260619 11:13:33.420614 25094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:33.420781 25094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:33.423385 25094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c1c5e2616794eb9a1582b4fc6339009: Sending DeleteTablet for 3 replicas of tablet ffe3678c721d4f17abdde3154faf5b04
I20260619 11:13:33.424168 25479 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe3678c721d4f17abdde3154faf5b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:33 UTC) from {username='slave'} at 127.0.0.1:45574
I20260619 11:13:33.424158 25348 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe3678c721d4f17abdde3154faf5b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:33 UTC) from {username='slave'} at 127.0.0.1:38048
I20260619 11:13:33.424659 25585 tablet_replica.cc:330] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: stopping tablet replica
I20260619 11:13:33.424687 25217 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe3678c721d4f17abdde3154faf5b04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:33 UTC) from {username='slave'} at 127.0.0.1:51832
I20260619 11:13:33.425009 25585 raft_consensus.cc:2229] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:33.425133 25586 tablet_replica.cc:330] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de: stopping tablet replica
I20260619 11:13:33.425303 25585 raft_consensus.cc:2258] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:33.425383 25586 raft_consensus.cc:2229] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:13:33.425693 25584 tablet_replica.cc:330] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: stopping tablet replica
I20260619 11:13:33.425724 25586 raft_consensus.cc:2258] T ffe3678c721d4f17abdde3154faf5b04 P 0a5887eb0232451c926e095c3e7065de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:33.425992 25584 raft_consensus.cc:2229] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:33.426039 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25155
I20260619 11:13:33.426277 25584 raft_consensus.cc:2258] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:33.428257 25585 ts_tablet_manager.cc:1905] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:33.429234 25584 ts_tablet_manager.cc:1905] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:33.431504 25585 ts_tablet_manager.cc:1918] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:33.431625 25585 log.cc:1196] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/ffe3678c721d4f17abdde3154faf5b04
I20260619 11:13:33.432025 25585 ts_tablet_manager.cc:1939] T ffe3678c721d4f17abdde3154faf5b04 P efaffc5e6fbf4f988c0a1109a25fe7a7: Deleting consensus metadata
I20260619 11:13:33.432334 25584 ts_tablet_manager.cc:1918] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:33.432430 25584 log.cc:1196] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/ffe3678c721d4f17abdde3154faf5b04
I20260619 11:13:33.432776 25584 ts_tablet_manager.cc:1939] T ffe3678c721d4f17abdde3154faf5b04 P d2263960078544e1a009dca61ae4dcc3: Deleting consensus metadata
I20260619 11:13:33.433316 25081 catalog_manager.cc:4841] TS efaffc5e6fbf4f988c0a1109a25fe7a7 (127.7.90.66:35253): tablet ffe3678c721d4f17abdde3154faf5b04 (table test-table [id=96e26bee6a744ca5ae73870b7381b618]) successfully deleted
I20260619 11:13:33.433928 25082 catalog_manager.cc:4841] TS d2263960078544e1a009dca61ae4dcc3 (127.7.90.67:44417): tablet ffe3678c721d4f17abdde3154faf5b04 (table test-table [id=96e26bee6a744ca5ae73870b7381b618]) successfully deleted
W20260619 11:13:33.435501 25082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:33997 (error 108)
I20260619 11:13:33.436177 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25286
I20260619 11:13:33.436645 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:33.436250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:33997 (local address 127.0.0.1:51856)
0619 11:13:33.436461 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:33.436541 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 11:13:33.436862 25082 catalog_manager.cc:4584] TS 0a5887eb0232451c926e095c3e7065de (127.7.90.65:33997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffe3678c721d4f17abdde3154faf5b04: Network error: Client connection negotiation failed: client connection to 127.7.90.65:33997: connect: Connection refused (error 111)
I20260619 11:13:33.442562 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25417
I20260619 11:13:33.447870 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25064
2026-06-19T11:13:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1011 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:33 dist-test-slave-f1t5 krb5kdc[25590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:33 dist-test-slave-f1t5 krb5kdc[25590](info): set up 2 sockets
Jun 19 11:13:33 dist-test-slave-f1t5 krb5kdc[25590](info): commencing operation
krb5kdc: starting...
W20260619 11:13:34.501499 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:34 dist-test-slave-f1t5 krb5kdc[25590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:13:34.614238 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35125
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:34.718536 25603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:34.718873 25603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:34.718955 25603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:34.722877 25603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:34.722947 25603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:34.722985 25603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:34.723042 25603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:34.723078 25603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:34.728150 25603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:34.729115 25603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:34.730134 25603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:34.738296 25608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:34.738404 25603 server_base.cc:1034] running on GCE node
W20260619 11:13:34.738272 25611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:34.738327 25609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:34.739054 25603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:34.740127 25603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:34.741343 25603 hybrid_clock.cc:648] HybridClock initialized: now 1781867614741316 us; error 39 us; skew 500 ppm
Jun 19 11:13:34 dist-test-slave-f1t5 krb5kdc[25590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:34.744177 25603 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:34.745021 25603 webserver.cc:460] Webserver started at http://127.7.90.126:45715/ using document root <none> and password file <none>
I20260619 11:13:34.745576 25603 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:34.745640 25603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:34.745806 25603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:34.747686 25603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "286fb189e637438d94715931ed4163b0"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "63b7216102f4b8830fbe214caa872b27"
server_key_iv: "2afd77b8b46ae3672c03a160ffe88c68"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.748171 25603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "286fb189e637438d94715931ed4163b0"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "63b7216102f4b8830fbe214caa872b27"
server_key_iv: "2afd77b8b46ae3672c03a160ffe88c68"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.751650 25603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:13:34.753932 25618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:34.755101 25603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:34.755240 25603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "286fb189e637438d94715931ed4163b0"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "63b7216102f4b8830fbe214caa872b27"
server_key_iv: "2afd77b8b46ae3672c03a160ffe88c68"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.755393 25603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:34.766669 25603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:34.769251 25603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:34.769443 25603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:34.776656 25603 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:35125
I20260619 11:13:34.776678 25670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:35125 every 8 connection(s)
I20260619 11:13:34.777551 25603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:34.779273 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25603
I20260619 11:13:34.779392 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:34.779683 7529 external_mini_cluster.cc:1428] Setting key 499d0b4b28de92a925940b6680ad010d
I20260619 11:13:34.780355 25671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:34.786643 25671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: Bootstrap starting.
Jun 19 11:13:34 dist-test-slave-f1t5 krb5kdc[25590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:34.788731 25671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:34.789614 25671 log.cc:824] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:34.791342 25671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: No bootstrap required, opened a new log
I20260619 11:13:34.793041 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:34.780964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.7.90.126:35125)
0619 11:13:34.781349 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:34.781364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:34.781396 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:13:34.782091 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:34.782106 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:34.782423 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:34.782694 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:34.783583 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:34.784525 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:34.785722 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:34.786157 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:34.786267 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:34.788678 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:34.788683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:34.788694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:34.788719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:34.790474 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:34.790899 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:34.790910 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:34.790914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:34.791013 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:34.791281 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:34.791286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:34.791288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:34.791690 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:34.791880 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:34.792138 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 11:13:34.792394 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
I20260619 11:13:34.794766 25671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } }
I20260619 11:13:34.795007 25671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:34.795113 25671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286fb189e637438d94715931ed4163b0, State: Initialized, Role: FOLLOWER
I20260619 11:13:34.795485 25671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } }
I20260619 11:13:34.795591 25671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:34.795634 25671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:34.795711 25671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:34.796418 25671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } }
I20260619 11:13:34.796674 25671 leader_election.cc:302] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 286fb189e637438d94715931ed4163b0; no voters:
I20260619 11:13:34.796881 25671 leader_election.cc:288] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:34.796996 25676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:34.797266 25676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [term 1 LEADER]: Becoming Leader. State: Replica: 286fb189e637438d94715931ed4163b0, State: Running, Role: LEADER
I20260619 11:13:34.797681 25676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } }
I20260619 11:13:34.798053 25671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:34.799206 25678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286fb189e637438d94715931ed4163b0. Latest consensus state: current_term: 1 leader_uuid: "286fb189e637438d94715931ed4163b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } } }
I20260619 11:13:34.799329 25678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:34.799706 25677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "286fb189e637438d94715931ed4163b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286fb189e637438d94715931ed4163b0" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35125 } } }
I20260619 11:13:34.800290 25677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0 [sys.catalog]: This master's current role is: LEADER
W20260619 11:13:34.802546 25691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:13:34.802606 25691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:13:34.802685 25685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:34.803390 25685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:34.808364 25685 catalog_manager.cc:1347] Generated new cluster ID: 64d85550026342c1b34f0011d65ad6a1
I20260619 11:13:34.808430 25685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:34.816789 25685 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:34.817912 25685 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:34.823691 25685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: Generated new TSK 0
I20260619 11:13:34.824270 25685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:13:34.855927 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:34.960029 25697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:34.960284 25697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:34.960359 25697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:34.964310 25697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:34.964391 25697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:34.964478 25697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:34.969475 25697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:34.970491 25697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:34.971447 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:34.980549 25702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:34.980561 25703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:34.980728 25697 server_base.cc:1034] running on GCE node
W20260619 11:13:34.980930 25705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:34.981281 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:34.981920 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:34.983101 25697 hybrid_clock.cc:648] HybridClock initialized: now 1781867614983087 us; error 26 us; skew 500 ppm
Jun 19 11:13:34 dist-test-slave-f1t5 krb5kdc[25590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:34.985730 25697 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:13:34.986497 25697 webserver.cc:460] Webserver started at http://127.7.90.65:44135/ using document root <none> and password file <none>
I20260619 11:13:34.986975 25697 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:34.987030 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:34.987182 25697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:34.989039 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "130f99e53de64b819e4941a7e4958875"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "4f199a43b9c566f957425528749bcf4f"
server_key_iv: "4ee319249aa426ab484ee07d0b96006a"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.989573 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "130f99e53de64b819e4941a7e4958875"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "4f199a43b9c566f957425528749bcf4f"
server_key_iv: "4ee319249aa426ab484ee07d0b96006a"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.992951 25697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:13:34.995239 25712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:34.996438 25697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:13:34.996551 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "130f99e53de64b819e4941a7e4958875"
format_stamp: "Formatted at 2026-06-19 11:13:34 on dist-test-slave-f1t5"
server_key: "4f199a43b9c566f957425528749bcf4f"
server_key_iv: "4ee319249aa426ab484ee07d0b96006a"
server_key_version: "encryptionkey@0"
I20260619 11:13:34.996646 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:35.009317 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:35.011914 25697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:35.012112 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:35.012636 25697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:35.013409 25697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:35.013454 25697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.013572 25697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:35.013613 25697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.022343 25697 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:40045
I20260619 11:13:35.022368 25825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:40045 every 8 connection(s)
I20260619 11:13:35.023140 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:35.031615 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25697
I20260619 11:13:35.031785 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
Jun 19 11:13:35 dist-test-slave-f1t5 krb5kdc[25590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867614, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:35.032222 7529 external_mini_cluster.cc:1428] Setting key 6533b06993ef4cd37d687f025eb1e565
I20260619 11:13:35.036868 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.024816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:37815 (local address 127.7.90.126:35125)
0619 11:13:35.024955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.024958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.025761 (+ 803us) server_negotiation.cc:407] Connection header received
0619 11:13:35.026648 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.026652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.026710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.026816 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:35.027903 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.028442 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.029629 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.029917 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.030042 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.032750 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:35.032755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:35.032758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:35.032790 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:35.034423 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.035114 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.035117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.035118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.035197 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.035638 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.035643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.035645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.035912 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:35.036024 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.036502 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.036658 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:35.037612 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.025116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35125 (local address 127.7.90.65:37815)
0619 11:13:35.025628 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.025668 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.026450 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.027006 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.027013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.027327 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:35.027755 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.027767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.028589 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.028592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.029472 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.029480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.030073 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.030077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.030240 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.030740 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:35.030755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:35.032567 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:35.034549 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.034558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.034575 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.035008 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.035295 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.035301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.035307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.035510 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.036027 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:35.036031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:35.036257 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.036426 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.036673 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260619 11:13:35.039031 25826 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35125
I20260619 11:13:35.039278 25826 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:35.039901 25826 heartbeater.cc:511] Master 127.7.90.126:35125 requested a full tablet report, sending...
I20260619 11:13:35.041437 25634 ts_manager.cc:194] Registered new tserver with Master: 130f99e53de64b819e4941a7e4958875 (127.7.90.65:40045)
I20260619 11:13:35.042690 25634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:37815
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:13:35.060057 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:35.168681 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:35.168918 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:35.168960 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:35.172775 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:35.172817 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:35.172874 25831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:35.178143 25831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:35.179016 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:35.179898 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:35.188902 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:35.189013 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:35.189150 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:35.189285 25831 server_base.cc:1034] running on GCE node
I20260619 11:13:35.189760 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:35.190380 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:35.191568 25831 hybrid_clock.cc:648] HybridClock initialized: now 1781867615191562 us; error 33 us; skew 500 ppm
Jun 19 11:13:35 dist-test-slave-f1t5 krb5kdc[25590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:35.194306 25831 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:13:35.195075 25831 webserver.cc:460] Webserver started at http://127.7.90.66:42043/ using document root <none> and password file <none>
I20260619 11:13:35.195609 25831 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:35.195668 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:35.195820 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:35.197777 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "088b8248c0e043ddb7e63735b1b11c72"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "06c7645c5ba72d436b600043e0beb485"
server_key_iv: "dd8e7771d512125578cf36c3db399269"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.198376 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "088b8248c0e043ddb7e63735b1b11c72"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "06c7645c5ba72d436b600043e0beb485"
server_key_iv: "dd8e7771d512125578cf36c3db399269"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.202198 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:13:35.204376 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.205626 25831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:35.205739 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "088b8248c0e043ddb7e63735b1b11c72"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "06c7645c5ba72d436b600043e0beb485"
server_key_iv: "dd8e7771d512125578cf36c3db399269"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.205837 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:35.223915 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:35.227023 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:35.227212 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:35.227738 25831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:35.228487 25831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:35.228533 25831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.228597 25831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:35.228634 25831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.237349 25831 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:33487
I20260619 11:13:35.237373 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:33487 every 8 connection(s)
I20260619 11:13:35.238173 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:35.246028 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25831
I20260619 11:13:35.246145 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:35.246351 7529 external_mini_cluster.cc:1428] Setting key 2ced4e76718d0769414a2a69ca949eaf
Jun 19 11:13:35 dist-test-slave-f1t5 krb5kdc[25590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:13:35.254822 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.239794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:42275 (local address 127.7.90.126:35125)
0619 11:13:35.239933 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.239938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.240716 (+ 778us) server_negotiation.cc:407] Connection header received
0619 11:13:35.241485 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.241488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.241550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.241618 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:35.242617 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.243213 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.244437 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.244680 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.244801 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.251355 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:35.251359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:35.251361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:35.251404 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:35.252773 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.253343 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.253345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.253347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.253427 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.253792 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.253796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.253798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.253972 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:35.254080 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.254535 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.254650 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:13:35.255630 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.240162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35125 (local address 127.7.90.66:42275)
0619 11:13:35.240569 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.240607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.241305 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.241745 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.241751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.242072 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:35.242496 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.242509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.243319 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.243322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.244176 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.244185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.244833 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.244835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.244973 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.249853 (+ 4880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:35.249869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:35.251195 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:35.252934 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.252940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.252953 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.253210 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.253551 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.253556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.253558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.253695 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.254077 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:35.254084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:35.254314 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.254495 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.254799 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260619 11:13:35.256700 25960 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:13:35.256990 25960 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:35.257563 25960 heartbeater.cc:511] Master 127.7.90.126:35125 requested a full tablet report, sending...
I20260619 11:13:35.258612 25634 ts_manager.cc:194] Registered new tserver with Master: 088b8248c0e043ddb7e63735b1b11c72 (127.7.90.66:33487)
I20260619 11:13:35.259219 25634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:42275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:13:35.270685 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:35.379163 25965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:35.379398 25965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:35.379439 25965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:35.383440 25965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:35.383491 25965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:35.383579 25965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:35.388797 25965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:35.389859 25965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:35.390859 25965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:35.400487 25971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:35.400555 25970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:35.400683 25973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:35.400707 25965 server_base.cc:1034] running on GCE node
I20260619 11:13:35.401337 25965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:35.402067 25965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:35.403250 25965 hybrid_clock.cc:648] HybridClock initialized: now 1781867615403242 us; error 42 us; skew 500 ppm
Jun 19 11:13:35 dist-test-slave-f1t5 krb5kdc[25590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:35.405999 25965 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:13:35.406838 25965 webserver.cc:460] Webserver started at http://127.7.90.67:32897/ using document root <none> and password file <none>
I20260619 11:13:35.407349 25965 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:35.407410 25965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:35.407573 25965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:35.409839 25965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "29ddaeee2f0441ccb2216999aa71b0fc"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "5b05f95d310f793463cb67f54f832108"
server_key_iv: "33a24ca2c1471ee8d25c9002fa010396"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.410382 25965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "29ddaeee2f0441ccb2216999aa71b0fc"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "5b05f95d310f793463cb67f54f832108"
server_key_iv: "33a24ca2c1471ee8d25c9002fa010396"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.414160 25965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 11:13:35.416512 25980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.417862 25965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:35.417989 25965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "29ddaeee2f0441ccb2216999aa71b0fc"
format_stamp: "Formatted at 2026-06-19 11:13:35 on dist-test-slave-f1t5"
server_key: "5b05f95d310f793463cb67f54f832108"
server_key_iv: "33a24ca2c1471ee8d25c9002fa010396"
server_key_version: "encryptionkey@0"
I20260619 11:13:35.418104 25965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:35.442642 25965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:35.446237 25965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:35.446481 25965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:35.447106 25965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:35.447983 25965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:35.448040 25965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.448112 25965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:35.448154 25965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:35.457244 25965 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:37301
I20260619 11:13:35.457317 26093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:37301 every 8 connection(s)
I20260619 11:13:35.458310 25965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:35.466367 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 25965
I20260619 11:13:35.466500 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:35.466773 7529 external_mini_cluster.cc:1428] Setting key 712fd3771b25531e49e14ddf65a90b22
Jun 19 11:13:35 dist-test-slave-f1t5 krb5kdc[25590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:35.471370 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.460207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:51967 (local address 127.7.90.126:35125)
0619 11:13:35.460354 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.460366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.461295 (+ 929us) server_negotiation.cc:407] Connection header received
0619 11:13:35.462208 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.462211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.462258 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.462339 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:35.463459 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.464005 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.465103 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.465328 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.465410 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.468181 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:35.468185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:35.468188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:35.468220 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:35.469655 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.470118 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.470121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.470122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.470200 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.470485 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.470487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.470488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.470659 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:35.470734 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.471158 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.471251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:13:35.472121 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.460493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35125 (local address 127.7.90.67:51967)
0619 11:13:35.461146 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.461186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.462026 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.462500 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.462507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.462853 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:35.463319 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.463333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.464110 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.464113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.464997 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.465006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.465455 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.465458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.465669 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.466260 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:35.466275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:35.468010 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:35.469755 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.469760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.469771 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.470024 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.470281 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:35.470284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:35.470285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:35.470406 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:35.470744 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:35.470748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:35.470964 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.471112 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.471343 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":113,"threads_started":1}
I20260619 11:13:35.473335 26094 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35125
I20260619 11:13:35.473598 26094 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:35.474022 26094 heartbeater.cc:511] Master 127.7.90.126:35125 requested a full tablet report, sending...
I20260619 11:13:35.474967 25634 ts_manager.cc:194] Registered new tserver with Master: 29ddaeee2f0441ccb2216999aa71b0fc (127.7.90.67:37301)
I20260619 11:13:35.475507 25634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:51967
I20260619 11:13:35.480677 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:35.489672 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.482637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.7.90.126:35125)
0619 11:13:35.482776 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.482780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.482927 (+ 147us) server_negotiation.cc:407] Connection header received
0619 11:13:35.483016 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.483018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.483054 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.483128 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:35.483761 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.484275 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.485171 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.485372 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.485448 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.486913 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:35.486916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:35.486918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:35.486935 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:35.488119 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.488563 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.488565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.488566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.488631 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:35.488917 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:35.488920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:35.488922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:35.489117 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:35.489227 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.489443 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.489512 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:13:35.492223 25634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:35.495277 25634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:35.509747 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.504177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.7.90.67:37301)
0619 11:13:35.504546 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.504557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.505192 (+ 635us) server_negotiation.cc:407] Connection header received
0619 11:13:35.505269 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.505274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.505454 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.505663 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:35.506072 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.507024 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.508340 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.509078 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.509195 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.509344 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.509482 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.509592 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":155,"threads_started":1}
I20260619 11:13:35.509807 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.504150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.7.90.65:40045)
0619 11:13:35.505002 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.505018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.505042 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:13:35.505116 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.505121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.505257 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.505418 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:35.505623 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.506623 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.508256 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.508936 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.509020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.509161 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.509574 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.509652 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":96,"threads_started":1}
I20260619 11:13:35.509845 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.503443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:40045 (local address 127.0.0.1:59106)
0619 11:13:35.504743 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.504756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.504851 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.505390 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.505393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.505410 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:35.505509 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.505537 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.506751 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.506754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.508123 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.508133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.509058 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.509062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.509195 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.509240 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.509424 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.509697 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":225,"thread_start_us":98,"threads_started":1}
I20260619 11:13:35.510350 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.503726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:37301 (local address 127.0.0.1:38288)
0619 11:13:35.505019 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.505031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.505105 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.505817 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.505821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.505842 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:35.505947 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.505952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.507201 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.507206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.508217 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.508230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.509218 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.509222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.509335 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.509358 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.509437 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.510215 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":5,"thread_start_us":111,"threads_started":1}
I20260619 11:13:35.510599 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.503412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:33487 (local address 127.0.0.1:38174)
0619 11:13:35.503740 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.503794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.503920 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.504954 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.504962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.504983 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:35.505128 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.505138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.506549 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.506554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.508124 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.508133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.510216 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.510219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.510333 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.510355 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.510421 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.510490 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":111,"thread_start_us":71,"threads_started":1}
I20260619 11:13:35.511366 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.504077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.7.90.66:33487)
0619 11:13:35.504449 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.504461 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.504485 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:13:35.504566 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.504573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.504787 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.505276 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:35.505437 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.506422 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.509228 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.510077 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.510961 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.511088 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.511136 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.511222 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":147,"threads_started":1}
I20260619 11:13:35.512809 25760 tablet_service.cc:1467] Processing CreateTablet for tablet d2a208096bd748eaab794cd15bc167e5 (DEFAULT_TABLE table=test-table [id=8d4d48b7fbda4131bbf084cd4de158ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:35.512938 26028 tablet_service.cc:1467] Processing CreateTablet for tablet d2a208096bd748eaab794cd15bc167e5 (DEFAULT_TABLE table=test-table [id=8d4d48b7fbda4131bbf084cd4de158ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:35.513844 25760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a208096bd748eaab794cd15bc167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:35.513959 26028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a208096bd748eaab794cd15bc167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:35.513868 25894 tablet_service.cc:1467] Processing CreateTablet for tablet d2a208096bd748eaab794cd15bc167e5 (DEFAULT_TABLE table=test-table [id=8d4d48b7fbda4131bbf084cd4de158ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:35.514726 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a208096bd748eaab794cd15bc167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:35.520349 26111 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Bootstrap starting.
I20260619 11:13:35.521297 26112 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Bootstrap starting.
I20260619 11:13:35.521914 26113 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Bootstrap starting.
I20260619 11:13:35.522693 26111 tablet_bootstrap.cc:654] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:35.523466 26112 tablet_bootstrap.cc:654] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:35.523766 26111 log.cc:824] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:35.524317 26113 tablet_bootstrap.cc:654] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:35.524775 26112 log.cc:824] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:35.525092 26113 log.cc:824] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:35.525928 26111 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: No bootstrap required, opened a new log
I20260619 11:13:35.526124 26111 ts_tablet_manager.cc:1397] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:13:35.526619 26112 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: No bootstrap required, opened a new log
I20260619 11:13:35.526813 26112 ts_tablet_manager.cc:1397] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:13:35.526985 26113 tablet_bootstrap.cc:492] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: No bootstrap required, opened a new log
I20260619 11:13:35.527160 26113 ts_tablet_manager.cc:1397] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:13:35.529673 26112 raft_consensus.cc:348] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.529947 26112 raft_consensus.cc:374] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:35.530019 26112 raft_consensus.cc:729] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ddaeee2f0441ccb2216999aa71b0fc, State: Initialized, Role: FOLLOWER
I20260619 11:13:35.530043 26111 raft_consensus.cc:348] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.530326 26111 raft_consensus.cc:374] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:35.530386 26111 raft_consensus.cc:729] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130f99e53de64b819e4941a7e4958875, State: Initialized, Role: FOLLOWER
I20260619 11:13:35.530510 26112 consensus_queue.cc:260] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.530803 26111 consensus_queue.cc:260] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.530901 26113 raft_consensus.cc:348] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.531183 26094 heartbeater.cc:503] Master 127.7.90.126:35125 was elected leader, sending a full tablet report...
I20260619 11:13:35.531199 26113 raft_consensus.cc:374] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:35.531258 26113 raft_consensus.cc:729] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088b8248c0e043ddb7e63735b1b11c72, State: Initialized, Role: FOLLOWER
I20260619 11:13:35.531553 26112 ts_tablet_manager.cc:1428] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:13:35.531666 26113 consensus_queue.cc:260] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.531785 26111 ts_tablet_manager.cc:1428] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:13:35.532119 25826 heartbeater.cc:503] Master 127.7.90.126:35125 was elected leader, sending a full tablet report...
I20260619 11:13:35.532459 26113 ts_tablet_manager.cc:1428] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:13:35.532742 25960 heartbeater.cc:503] Master 127.7.90.126:35125 was elected leader, sending a full tablet report...
W20260619 11:13:35.709331 26095 tablet.cc:2369] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:35.736733 26119 raft_consensus.cc:482] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:35.736987 26119 raft_consensus.cc:504] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.738361 26119 leader_election.cc:288] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130f99e53de64b819e4941a7e4958875 (127.7.90.65:40045), 29ddaeee2f0441ccb2216999aa71b0fc (127.7.90.67:37301)
W20260619 11:13:35.739219 25961 tablet.cc:2369] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:35.742887 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.738562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:40045 (local address 127.7.90.66:54099)
0619 11:13:35.738683 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.738702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.738811 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.739115 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.739118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.739136 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:35.739264 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.739270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.740257 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.740260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.741492 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.741502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.742458 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.742461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.742564 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.742591 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.742669 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.742745 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":6}
I20260619 11:13:35.742952 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.738641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:54099 (local address 127.7.90.65:40045)
0619 11:13:35.738772 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.738778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.738795 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:13:35.738943 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.738947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.739000 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.739102 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:35.739426 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.740146 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.741623 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.742325 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.742473 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.742560 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.742703 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.742815 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:13:35.742988 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.738876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:58583 (local address 127.7.90.67:37301)
0619 11:13:35.739024 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.739031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.739049 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:13:35.739231 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.739237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.739296 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.739346 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:35.739746 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.740409 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.741793 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.742488 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.742596 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.742679 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.742804 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.742874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:13:35.743022 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.738604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:37301 (local address 127.7.90.66:58583)
0619 11:13:35.738971 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.738983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:35.739092 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:35.739482 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:35.739484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:35.739499 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:35.739611 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.739616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.740504 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.740507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.741681 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:35.741687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.742591 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.742594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:35.742694 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.742712 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:13:35.742776 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:13:35.742831 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":25,"thread_start_us":68,"threads_started":1}
I20260619 11:13:35.743670 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a208096bd748eaab794cd15bc167e5" candidate_uuid: "088b8248c0e043ddb7e63735b1b11c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" is_pre_election: true
I20260619 11:13:35.743672 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a208096bd748eaab794cd15bc167e5" candidate_uuid: "088b8248c0e043ddb7e63735b1b11c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130f99e53de64b819e4941a7e4958875" is_pre_election: true
I20260619 11:13:35.743980 25780 raft_consensus.cc:2454] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088b8248c0e043ddb7e63735b1b11c72 in term 0.
I20260619 11:13:35.743980 26048 raft_consensus.cc:2454] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088b8248c0e043ddb7e63735b1b11c72 in term 0.
I20260619 11:13:35.744485 25847 leader_election.cc:302] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 088b8248c0e043ddb7e63735b1b11c72, 130f99e53de64b819e4941a7e4958875, 29ddaeee2f0441ccb2216999aa71b0fc; no voters:
I20260619 11:13:35.744753 26119 raft_consensus.cc:2781] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:35.744844 26119 raft_consensus.cc:482] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:35.744925 26119 raft_consensus.cc:3037] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:35.746042 26119 raft_consensus.cc:504] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.746398 26119 leader_election.cc:288] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [CANDIDATE]: Term 1 election: Requested vote from peers 130f99e53de64b819e4941a7e4958875 (127.7.90.65:40045), 29ddaeee2f0441ccb2216999aa71b0fc (127.7.90.67:37301)
I20260619 11:13:35.746745 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a208096bd748eaab794cd15bc167e5" candidate_uuid: "088b8248c0e043ddb7e63735b1b11c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130f99e53de64b819e4941a7e4958875"
I20260619 11:13:35.746769 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a208096bd748eaab794cd15bc167e5" candidate_uuid: "088b8248c0e043ddb7e63735b1b11c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ddaeee2f0441ccb2216999aa71b0fc"
I20260619 11:13:35.746851 25780 raft_consensus.cc:3037] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:35.746902 26048 raft_consensus.cc:3037] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:35.747879 25780 raft_consensus.cc:2454] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088b8248c0e043ddb7e63735b1b11c72 in term 1.
I20260619 11:13:35.747884 26048 raft_consensus.cc:2454] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088b8248c0e043ddb7e63735b1b11c72 in term 1.
I20260619 11:13:35.748165 25847 leader_election.cc:302] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088b8248c0e043ddb7e63735b1b11c72, 130f99e53de64b819e4941a7e4958875; no voters:
I20260619 11:13:35.748342 26119 raft_consensus.cc:2781] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:35.748580 26119 raft_consensus.cc:686] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 1 LEADER]: Becoming Leader. State: Replica: 088b8248c0e043ddb7e63735b1b11c72, State: Running, Role: LEADER
I20260619 11:13:35.748911 26119 consensus_queue.cc:237] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } }
I20260619 11:13:35.751844 25635 catalog_manager.cc:5495] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088b8248c0e043ddb7e63735b1b11c72 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "088b8248c0e043ddb7e63735b1b11c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b8248c0e043ddb7e63735b1b11c72" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 33487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 } health_report { overall_health: UNKNOWN } } }
W20260619 11:13:35.774427 25827 tablet.cc:2369] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:35.802426 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.798581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.7.90.66:33487)
0619 11:13:35.798738 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:35.798744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:35.798784 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:13:35.798854 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:35.798856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:35.798911 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:35.798999 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:35.799221 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.799825 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.800651 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:35.800872 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:35.800960 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:35.801317 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:35.801999 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:35.802216 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:35.802264 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:13:35.802297 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:35.808418 26048 raft_consensus.cc:1261] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 1 FOLLOWER]: Refusing update from remote peer 088b8248c0e043ddb7e63735b1b11c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:35.808441 25780 raft_consensus.cc:1261] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 1 FOLLOWER]: Refusing update from remote peer 088b8248c0e043ddb7e63735b1b11c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:35.809144 26119 consensus_queue.cc:1035] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ddaeee2f0441ccb2216999aa71b0fc" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:35.809427 26121 consensus_queue.cc:1035] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130f99e53de64b819e4941a7e4958875" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 40045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:35.817535 26124 mvcc.cc:205] Tried to move back new op lower bound from 7298529754343440384 to 7298529754110009344. Current Snapshot: MvccSnapshot[applied={T|T < 7298529754343440384}]
I20260619 11:13:35.817550 26125 mvcc.cc:205] Tried to move back new op lower bound from 7298529754343440384 to 7298529754110009344. Current Snapshot: MvccSnapshot[applied={T|T < 7298529754343440384}]
I20260619 11:13:35.825464 25635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:35.825659 25635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:35.827935 25635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 286fb189e637438d94715931ed4163b0: Sending DeleteTablet for 3 replicas of tablet d2a208096bd748eaab794cd15bc167e5
I20260619 11:13:35.828639 25894 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a208096bd748eaab794cd15bc167e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:35 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:38174
I20260619 11:13:35.828804 26028 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a208096bd748eaab794cd15bc167e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:35 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:38288
I20260619 11:13:35.828807 25760 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a208096bd748eaab794cd15bc167e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:35 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:59106
I20260619 11:13:35.829264 26134 tablet_replica.cc:330] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: stopping tablet replica
I20260619 11:13:35.829264 26136 tablet_replica.cc:330] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875: stopping tablet replica
I20260619 11:13:35.829604 26134 raft_consensus.cc:2229] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:13:35.829734 26136 raft_consensus.cc:2229] T d2a208096bd748eaab794cd15bc167e5 P 130f99e53de64b819e4941a7e4958875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:35.830003 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25697
I20260619 11:13:35.830031 26134 raft_consensus.cc:2258] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:35.832111 26135 tablet_replica.cc:330] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: stopping tablet replica
I20260619 11:13:35.832495 26135 raft_consensus.cc:2229] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:35.832926 26135 raft_consensus.cc:2258] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:35.833318 26134 ts_tablet_manager.cc:1905] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:35.834355 26135 ts_tablet_manager.cc:1905] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:35.836345 26134 ts_tablet_manager.cc:1918] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:35.836522 26134 log.cc:1196] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/d2a208096bd748eaab794cd15bc167e5
I20260619 11:13:35.836921 26134 ts_tablet_manager.cc:1939] T d2a208096bd748eaab794cd15bc167e5 P 088b8248c0e043ddb7e63735b1b11c72: Deleting consensus metadata
I20260619 11:13:35.837167 26135 ts_tablet_manager.cc:1918] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:35.837313 26135 log.cc:1196] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/d2a208096bd748eaab794cd15bc167e5
I20260619 11:13:35.837714 26135 ts_tablet_manager.cc:1939] T d2a208096bd748eaab794cd15bc167e5 P 29ddaeee2f0441ccb2216999aa71b0fc: Deleting consensus metadata
I20260619 11:13:35.838362 25621 catalog_manager.cc:4841] TS 088b8248c0e043ddb7e63735b1b11c72 (127.7.90.66:33487): tablet d2a208096bd748eaab794cd15bc167e5 (table test-table [id=8d4d48b7fbda4131bbf084cd4de158ef]) successfully deleted
W20260619 11:13:35.838665 25619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:40045 (error 108)
I20260619 11:13:35.838928 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25831
I20260619 11:13:35.840070 25622 catalog_manager.cc:4841] TS 29ddaeee2f0441ccb2216999aa71b0fc (127.7.90.67:37301): tablet d2a208096bd748eaab794cd15bc167e5 (table test-table [id=8d4d48b7fbda4131bbf084cd4de158ef]) successfully deleted
I20260619 11:13:35.841199 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:35.840929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:40045 (local address 127.0.0.1:59110)
0619 11:13:35.841061 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:35.841129 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 11:13:35.841444 25619 catalog_manager.cc:4584] TS 130f99e53de64b819e4941a7e4958875 (127.7.90.65:40045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2a208096bd748eaab794cd15bc167e5: Network error: Client connection negotiation failed: client connection to 127.7.90.65:40045: connect: Connection refused (error 111)
I20260619 11:13:35.844895 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25965
I20260619 11:13:35.853226 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 25603
2026-06-19T11:13:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2407 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T11:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:35Z Disabled control of system clock
I20260619 11:13:35.884168 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40969
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:35.995092 26141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:35.995347 26141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:35.995383 26141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:35.999368 26141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:35.999421 26141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:35.999439 26141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:35.999459 26141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:35.999476 26141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:36.004148 26141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:36.005090 26141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:36.006407 26141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:36.014288 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.014389 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.014333 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:36.014674 26141 server_base.cc:1034] running on GCE node
I20260619 11:13:36.015246 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:36.016274 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:36.017462 26141 hybrid_clock.cc:648] HybridClock initialized: now 1781867616017454 us; error 33 us; skew 500 ppm
I20260619 11:13:36.019203 26141 webserver.cc:460] Webserver started at http://127.7.90.126:43637/ using document root <none> and password file <none>
I20260619 11:13:36.019742 26141 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:36.019831 26141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:36.020051 26141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:36.021983 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "2325a7fdf9ee453d8ce04d14dfd53da1"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "5c96f373aecf5aaf42a1c70a569de5e2"
server_key_iv: "029867d02b7cc268191d690cec3d8409"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.022513 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "2325a7fdf9ee453d8ce04d14dfd53da1"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "5c96f373aecf5aaf42a1c70a569de5e2"
server_key_iv: "029867d02b7cc268191d690cec3d8409"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.025916 26141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:36.028203 26155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.029307 26141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:36.029459 26141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "2325a7fdf9ee453d8ce04d14dfd53da1"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "5c96f373aecf5aaf42a1c70a569de5e2"
server_key_iv: "029867d02b7cc268191d690cec3d8409"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.029592 26141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:36.044905 26141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:36.045563 26141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:36.045763 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:36.052434 26141 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40969
I20260619 11:13:36.052449 26207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40969 every 8 connection(s)
I20260619 11:13:36.053303 26141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:36.056304 26208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:36.059959 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26141
I20260619 11:13:36.060050 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:36.060310 7529 external_mini_cluster.cc:1428] Setting key 76bcd95984e57085688bed207cb7cfc8
I20260619 11:13:36.062187 26208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: Bootstrap starting.
I20260619 11:13:36.064850 26208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:36.065738 26208 log.cc:824] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:36.067730 26208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: No bootstrap required, opened a new log
I20260619 11:13:36.070952 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.061718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.7.90.126:40969)
0619 11:13:36.062359 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.062381 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.062427 (+ 46us) server_negotiation.cc:407] Connection header received
0619 11:13:36.063284 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.063309 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.063800 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.064145 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:36.064573 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.066104 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.067070 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.067723 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.067913 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.068686 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:36.068692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:36.068707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:36.069209 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:36.069273 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:36.069298 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:36.069405 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.069566 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.069831 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":100,"threads_started":1}
I20260619 11:13:36.071600 26208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } }
I20260619 11:13:36.071861 26208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:36.071990 26208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2325a7fdf9ee453d8ce04d14dfd53da1, State: Initialized, Role: FOLLOWER
I20260619 11:13:36.072475 26208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } }
I20260619 11:13:36.072599 26208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:36.072685 26208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:36.072809 26208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:36.073855 26208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } }
I20260619 11:13:36.074187 26208 leader_election.cc:302] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2325a7fdf9ee453d8ce04d14dfd53da1; no voters:
I20260619 11:13:36.074482 26208 leader_election.cc:288] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:36.074615 26213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:36.074847 26213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [term 1 LEADER]: Becoming Leader. State: Replica: 2325a7fdf9ee453d8ce04d14dfd53da1, State: Running, Role: LEADER
I20260619 11:13:36.075167 26213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } }
I20260619 11:13:36.075660 26208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:36.077026 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } } }
I20260619 11:13:36.077034 26215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2325a7fdf9ee453d8ce04d14dfd53da1. Latest consensus state: current_term: 1 leader_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2325a7fdf9ee453d8ce04d14dfd53da1" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40969 } } }
I20260619 11:13:36.077196 26215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:36.077481 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:36.078163 26223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:36.080571 26223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:36.085605 26223 catalog_manager.cc:1347] Generated new cluster ID: d8f4414b453445e5b57ab69f67572750
I20260619 11:13:36.085664 26223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:36.103032 26223 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:36.104143 26223 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:36.112468 26223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: Generated new TSK 0
I20260619 11:13:36.113186 26223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:36.121323 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--builtin_ntp_servers=127.7.90.84:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:36.230851 26232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:36.231099 26232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:36.231139 26232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:36.235196 26232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:36.235255 26232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:36.235311 26232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:36.239946 26232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:36.240882 26232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:36.242245 26232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:36.251556 26238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.251564 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:36.251684 26232 server_base.cc:1034] running on GCE node
W20260619 11:13:36.251621 26237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:36.252261 26232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:36.252856 26232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:36.254035 26232 hybrid_clock.cc:648] HybridClock initialized: now 1781867616254002 us; error 44 us; skew 500 ppm
I20260619 11:13:36.255674 26232 webserver.cc:460] Webserver started at http://127.7.90.65:46009/ using document root <none> and password file <none>
I20260619 11:13:36.256217 26232 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:36.256312 26232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:36.256525 26232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:36.258502 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "74b71844cdbc49cd8087f644f92fa798"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "2d89f810cfb81cb7489231bef1b1afd6"
server_key_iv: "b3108827dee3e3041bfd080d5dedaa23"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.259035 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "74b71844cdbc49cd8087f644f92fa798"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "2d89f810cfb81cb7489231bef1b1afd6"
server_key_iv: "b3108827dee3e3041bfd080d5dedaa23"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.262391 26232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:36.264658 26246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.265872 26232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:36.266000 26232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "74b71844cdbc49cd8087f644f92fa798"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "2d89f810cfb81cb7489231bef1b1afd6"
server_key_iv: "b3108827dee3e3041bfd080d5dedaa23"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.266105 26232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:36.279508 26232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:36.280095 26232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:36.280324 26232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:36.280867 26232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:36.281842 26232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:36.281921 26232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.281991 26232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:36.282040 26232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.290589 26232 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:43797
I20260619 11:13:36.290608 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:43797 every 8 connection(s)
I20260619 11:13:36.291414 26232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:36.297274 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26232
I20260619 11:13:36.297431 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:13:36.297792 7529 external_mini_cluster.cc:1428] Setting key 07a3d23ae592369d62b81b94db9b85fc
I20260619 11:13:36.299882 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--builtin_ntp_servers=127.7.90.84:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:36.300566 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.293290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:52389 (local address 127.7.90.126:40969)
0619 11:13:36.293451 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.293456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.294445 (+ 989us) server_negotiation.cc:407] Connection header received
0619 11:13:36.295780 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.295783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.295852 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.295924 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:36.296744 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.297327 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.298553 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.298806 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.298889 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.299861 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:36.299866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:36.299868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:36.299882 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:36.299922 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:36.299935 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:36.300024 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.300265 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.300333 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:13:36.301237 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.293680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40969 (local address 127.7.90.65:52389)
0619 11:13:36.294298 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.294338 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.295597 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.296084 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.296090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.296167 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:36.296611 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.296623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.297498 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.297504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.298439 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.298448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.298950 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.298953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.299096 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.299626 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:36.299627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:36.299662 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:36.299713 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:36.300074 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:36.300086 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:36.300093 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.300237 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.300450 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260619 11:13:36.302520 26360 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40969
I20260619 11:13:36.302749 26360 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:36.303347 26360 heartbeater.cc:511] Master 127.7.90.126:40969 requested a full tablet report, sending...
I20260619 11:13:36.304699 26168 ts_manager.cc:194] Registered new tserver with Master: 74b71844cdbc49cd8087f644f92fa798 (127.7.90.65:43797)
I20260619 11:13:36.305891 26168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.65:52389
W20260619 11:13:36.411324 26363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:36.411712 26363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:36.411814 26363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:36.418500 26363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:36.418562 26363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:36.418654 26363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:36.423241 26363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:36.424146 26363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:36.425376 26363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:36.434485 26369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.434496 26371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.434496 26368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:36.434821 26363 server_base.cc:1034] running on GCE node
I20260619 11:13:36.435217 26363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:36.435822 26363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:36.437033 26363 hybrid_clock.cc:648] HybridClock initialized: now 1781867616437020 us; error 27 us; skew 500 ppm
I20260619 11:13:36.438766 26363 webserver.cc:460] Webserver started at http://127.7.90.66:39715/ using document root <none> and password file <none>
I20260619 11:13:36.439322 26363 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:36.439411 26363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:36.439611 26363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:36.441498 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3b0a87b223712213fb91f11a98069e9f"
server_key_iv: "499ec32e4c5c9db86d06ffea206f7315"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.442093 26363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3b0a87b223712213fb91f11a98069e9f"
server_key_iv: "499ec32e4c5c9db86d06ffea206f7315"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.445498 26363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:13:36.447752 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.448894 26363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:13:36.449008 26363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3b0a87b223712213fb91f11a98069e9f"
server_key_iv: "499ec32e4c5c9db86d06ffea206f7315"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.449108 26363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:36.473641 26363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:36.474313 26363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:36.474500 26363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:36.475071 26363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:36.476145 26363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:36.476202 26363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.476241 26363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:36.476257 26363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.484774 26363 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:45913
I20260619 11:13:36.484817 26490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:45913 every 8 connection(s)
I20260619 11:13:36.485630 26363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:36.486558 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26363
I20260619 11:13:36.486681 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:36.486996 7529 external_mini_cluster.cc:1428] Setting key 1120ad98095b0839d1bbdb30b22cb4b5
I20260619 11:13:36.489056 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--builtin_ntp_servers=127.7.90.84:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:36.496191 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.487515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:53249 (local address 127.7.90.126:40969)
0619 11:13:36.487653 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.487658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.488789 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 11:13:36.490428 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.490431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.490477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.490545 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:36.491668 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.492174 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.494070 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.494283 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.494373 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.495736 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:36.495739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:36.495741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:36.495753 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:36.495771 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:36.495778 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:36.495835 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.496029 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.496078 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:36.497485 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.487925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40969 (local address 127.7.90.66:53249)
0619 11:13:36.488655 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.488695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.490158 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.490732 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.490743 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.490861 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:36.491541 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.491557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.492298 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.492305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.493930 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.493946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.494462 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.494469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.494719 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.495402 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:36.495405 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:36.495470 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:36.495563 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:36.495862 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:36.495873 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:36.495879 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.496029 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.496358 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260619 11:13:36.498675 26491 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40969
I20260619 11:13:36.498934 26491 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:36.499593 26491 heartbeater.cc:511] Master 127.7.90.126:40969 requested a full tablet report, sending...
I20260619 11:13:36.500715 26168 ts_manager.cc:194] Registered new tserver with Master: 94e4363b3e9e47fc85a3ba55e19c1ed9 (127.7.90.66:45913)
I20260619 11:13:36.501267 26168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.66:53249
W20260619 11:13:36.601137 26494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:36.601364 26494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:36.601432 26494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:36.605422 26494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:36.605474 26494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:36.605584 26494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:36.610194 26494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:36.611104 26494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:36.612485 26494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:36.621569 26500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.621515 26502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:36.621526 26499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:36.621767 26494 server_base.cc:1034] running on GCE node
I20260619 11:13:36.622195 26494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:36.622774 26494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:36.623929 26494 hybrid_clock.cc:648] HybridClock initialized: now 1781867616623915 us; error 29 us; skew 500 ppm
I20260619 11:13:36.625784 26494 webserver.cc:460] Webserver started at http://127.7.90.67:34367/ using document root <none> and password file <none>
I20260619 11:13:36.626313 26494 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:36.626372 26494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:36.626602 26494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:36.628492 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "0923482d55ec4ce6830b7e3a34a60758"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3ae2782386dbb7d7d6e549f836e357d3"
server_key_iv: "47899a4388fe582c42b8ee6fe3620a47"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.629045 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "0923482d55ec4ce6830b7e3a34a60758"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3ae2782386dbb7d7d6e549f836e357d3"
server_key_iv: "47899a4388fe582c42b8ee6fe3620a47"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.632527 26494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:13:36.634902 26508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.636013 26494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:13:36.636132 26494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "0923482d55ec4ce6830b7e3a34a60758"
format_stamp: "Formatted at 2026-06-19 11:13:36 on dist-test-slave-f1t5"
server_key: "3ae2782386dbb7d7d6e549f836e357d3"
server_key_iv: "47899a4388fe582c42b8ee6fe3620a47"
server_key_version: "encryptionkey@0"
I20260619 11:13:36.636225 26494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:36.647346 26494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:36.647931 26494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:36.648123 26494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:36.648625 26494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:36.649446 26494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:36.649513 26494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.649634 26494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:36.649695 26494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:36.657927 26494 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44423
I20260619 11:13:36.657995 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44423 every 8 connection(s)
I20260619 11:13:36.658754 26494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:36.665424 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26494
I20260619 11:13:36.665560 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:36.665807 7529 external_mini_cluster.cc:1428] Setting key 10c85209acf19dfdfccf63d21cc97df9
I20260619 11:13:36.667515 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.660419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:57385 (local address 127.7.90.126:40969)
0619 11:13:36.660578 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.660582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.661546 (+ 964us) server_negotiation.cc:407] Connection header received
0619 11:13:36.662865 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.662868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.662914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.662989 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:36.663779 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.664294 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.665482 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.665805 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.665921 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.667026 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:36.667029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:36.667032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:36.667046 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:36.667071 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:36.667084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:36.667160 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.667329 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.667384 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 11:13:36.668391 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.660791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:40969 (local address 127.7.90.67:57385)
0619 11:13:36.661383 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.661419 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.662658 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.663143 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.663150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.663229 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:36.663662 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.663672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.664394 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.664396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.665354 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.665366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.665969 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.665974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.666130 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.666709 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:36.666711 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:36.666784 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:36.666859 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:36.667170 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:36.667178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:36.667184 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.667313 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.667519 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260619 11:13:36.669423 26622 heartbeater.cc:347] Connected to a master server at 127.7.90.126:40969
I20260619 11:13:36.669695 26622 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:36.670303 26622 heartbeater.cc:511] Master 127.7.90.126:40969 requested a full tablet report, sending...
I20260619 11:13:36.671277 26168 ts_manager.cc:194] Registered new tserver with Master: 0923482d55ec4ce6830b7e3a34a60758 (127.7.90.67:44423)
I20260619 11:13:36.671836 26168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.67:57385
I20260619 11:13:36.679483 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:36.685312 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.681306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.7.90.126:40969)
0619 11:13:36.681397 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.681401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.681699 (+ 298us) server_negotiation.cc:407] Connection header received
0619 11:13:36.681823 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.681826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.681870 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.681952 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:36.682231 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.682718 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.683580 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.683793 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.683897 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.684809 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:36.684813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:36.684818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:36.684833 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:36.684849 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:36.684857 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:36.684980 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.685138 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.685175 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:13:36.687898 26168 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:36.690929 26168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:36.704164 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.698764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44423 (local address 127.0.0.1:53592)
0619 11:13:36.699278 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.699331 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.699469 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.700230 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.700236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.700260 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:36.700409 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.700416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.701701 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.701706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.702618 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.702628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.703590 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.703593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.703710 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.703747 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.703890 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.704016 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"spinlock_wait_cycles":896,"thread_start_us":71,"threads_started":1}
I20260619 11:13:36.704255 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.698903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.7.90.67:44423)
0619 11:13:36.699751 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.699764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.699790 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:13:36.699864 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.699872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.700086 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.700243 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:36.700544 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.701601 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.702759 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.703463 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.703578 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.703757 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.703920 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.704090 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":106,"threads_started":1}
I20260619 11:13:36.705060 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.699453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45913 (local address 127.0.0.1:33084)
0619 11:13:36.700082 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.700095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.700179 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.700903 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.700909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.700928 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:36.701117 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.701124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.702585 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.702589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.703641 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.703651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.704677 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.704684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.704783 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.704807 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.704873 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.704939 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":76,"thread_start_us":89,"threads_started":1}
I20260619 11:13:36.705169 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.699682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.7.90.66:45913)
0619 11:13:36.700445 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.700459 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.700484 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:13:36.700559 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.700564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.700737 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.700894 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:36.701258 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.702447 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.703769 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.704570 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.704678 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.704864 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.704925 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.705012 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":135,"threads_started":1}
I20260619 11:13:36.706106 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.698943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43797 (local address 127.0.0.1:50116)
0619 11:13:36.699278 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.699327 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.699465 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.700924 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.700926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.700938 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:36.701155 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.701166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.702585 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.702589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.704674 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.704684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.705717 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.705720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.705812 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.705834 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.705924 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.705986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":202,"spinlock_wait_cycles":1792,"thread_start_us":80,"threads_started":1}
I20260619 11:13:36.706702 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.699858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.7.90.65:43797)
0619 11:13:36.700552 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.700561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.700580 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:13:36.700637 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.700641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.700774 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.701113 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:36.701262 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.702447 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.704799 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.705587 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.705691 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.705853 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.706433 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.706536 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":133,"threads_started":1}
I20260619 11:13:36.707638 26556 tablet_service.cc:1467] Processing CreateTablet for tablet a058905995ff4070915bb25faaf819a6 (DEFAULT_TABLE table=test-table [id=d7692a3062584231ac487336f50d5eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:36.708256 26425 tablet_service.cc:1467] Processing CreateTablet for tablet a058905995ff4070915bb25faaf819a6 (DEFAULT_TABLE table=test-table [id=d7692a3062584231ac487336f50d5eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:36.708729 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a058905995ff4070915bb25faaf819a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:36.708796 26294 tablet_service.cc:1467] Processing CreateTablet for tablet a058905995ff4070915bb25faaf819a6 (DEFAULT_TABLE table=test-table [id=d7692a3062584231ac487336f50d5eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:36.709209 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a058905995ff4070915bb25faaf819a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:36.709597 26294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a058905995ff4070915bb25faaf819a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:36.715185 26639 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Bootstrap starting.
I20260619 11:13:36.716535 26641 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Bootstrap starting.
I20260619 11:13:36.716594 26640 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Bootstrap starting.
I20260619 11:13:36.717255 26639 tablet_bootstrap.cc:654] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:36.718204 26639 log.cc:824] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:36.718623 26640 tablet_bootstrap.cc:654] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:36.718693 26641 tablet_bootstrap.cc:654] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:36.719422 26641 log.cc:824] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:36.719422 26640 log.cc:824] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:36.720475 26639 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: No bootstrap required, opened a new log
I20260619 11:13:36.720670 26639 ts_tablet_manager.cc:1397] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:13:36.721182 26641 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: No bootstrap required, opened a new log
I20260619 11:13:36.721355 26641 ts_tablet_manager.cc:1397] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:13:36.721829 26640 tablet_bootstrap.cc:492] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: No bootstrap required, opened a new log
I20260619 11:13:36.722028 26640 ts_tablet_manager.cc:1397] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:13:36.724079 26639 raft_consensus.cc:348] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.724311 26639 raft_consensus.cc:374] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:36.724381 26639 raft_consensus.cc:729] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b71844cdbc49cd8087f644f92fa798, State: Initialized, Role: FOLLOWER
I20260619 11:13:36.724808 26639 consensus_queue.cc:260] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.725378 26360 heartbeater.cc:503] Master 127.7.90.126:40969 was elected leader, sending a full tablet report...
I20260619 11:13:36.725494 26641 raft_consensus.cc:348] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.725754 26641 raft_consensus.cc:374] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:36.725759 26639 ts_tablet_manager.cc:1428] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:13:36.725807 26641 raft_consensus.cc:729] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e4363b3e9e47fc85a3ba55e19c1ed9, State: Initialized, Role: FOLLOWER
I20260619 11:13:36.726217 26641 consensus_queue.cc:260] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.726311 26640 raft_consensus.cc:348] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.726615 26640 raft_consensus.cc:374] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:36.726706 26640 raft_consensus.cc:729] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0923482d55ec4ce6830b7e3a34a60758, State: Initialized, Role: FOLLOWER
I20260619 11:13:36.726897 26491 heartbeater.cc:503] Master 127.7.90.126:40969 was elected leader, sending a full tablet report...
I20260619 11:13:36.727174 26641 ts_tablet_manager.cc:1428] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:13:36.727164 26640 consensus_queue.cc:260] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.728024 26622 heartbeater.cc:503] Master 127.7.90.126:40969 was elected leader, sending a full tablet report...
I20260619 11:13:36.728509 26640 ts_tablet_manager.cc:1428] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 11:13:36.736449 26492 tablet.cc:2369] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:13:36.792543 26361 tablet.cc:2369] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:13:36.909773 26623 tablet.cc:2369] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:36.940982 26646 raft_consensus.cc:482] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:36.941218 26646 raft_consensus.cc:504] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.942576 26646 leader_election.cc:288] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0923482d55ec4ce6830b7e3a34a60758 (127.7.90.67:44423), 74b71844cdbc49cd8087f644f92fa798 (127.7.90.65:43797)
I20260619 11:13:36.947530 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.943044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:50107 (local address 127.7.90.65:43797)
0619 11:13:36.943173 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.943178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.943273 (+ 95us) server_negotiation.cc:407] Connection header received
0619 11:13:36.943435 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.943437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.943479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.943569 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:36.943802 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.944513 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946240 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.946854 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.947029 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.947332 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.947422 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:13:36.947557 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.942833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:44919 (local address 127.7.90.67:44423)
0619 11:13:36.942971 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.942978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.942997 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:13:36.943132 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.943136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.943191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.943276 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:36.943598 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.944429 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946240 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.946854 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.947029 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.947377 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.947453 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:13:36.947544 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.942829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43797 (local address 127.7.90.66:50107)
0619 11:13:36.943176 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.943188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.943314 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.943581 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.943584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.943608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:36.943698 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.943702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.944602 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.944605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.946126 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.946132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.946990 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.947170 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.947226 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.947306 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.947390 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":190,"thread_start_us":133,"threads_started":1}
I20260619 11:13:36.947969 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.942749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44423 (local address 127.7.90.66:44919)
0619 11:13:36.942873 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:36.942889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:36.942984 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:36.943307 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:36.943310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:36.943323 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:36.943470 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.943476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.944537 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.944540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.946118 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:36.946129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.946990 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.946994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:36.947180 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.947220 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 11:13:36.947298 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:13:36.947889 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":718}
I20260619 11:13:36.948329 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a058905995ff4070915bb25faaf819a6" candidate_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b71844cdbc49cd8087f644f92fa798" is_pre_election: true
I20260619 11:13:36.948664 26314 raft_consensus.cc:2454] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e4363b3e9e47fc85a3ba55e19c1ed9 in term 0.
I20260619 11:13:36.948652 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a058905995ff4070915bb25faaf819a6" candidate_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0923482d55ec4ce6830b7e3a34a60758" is_pre_election: true
I20260619 11:13:36.948948 26576 raft_consensus.cc:2454] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e4363b3e9e47fc85a3ba55e19c1ed9 in term 0.
I20260619 11:13:36.949177 26381 leader_election.cc:302] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b71844cdbc49cd8087f644f92fa798, 94e4363b3e9e47fc85a3ba55e19c1ed9; no voters:
I20260619 11:13:36.949479 26646 raft_consensus.cc:2781] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:36.949586 26646 raft_consensus.cc:482] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:36.949656 26646 raft_consensus.cc:3037] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:36.950656 26646 raft_consensus.cc:504] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.951040 26646 leader_election.cc:288] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [CANDIDATE]: Term 1 election: Requested vote from peers 0923482d55ec4ce6830b7e3a34a60758 (127.7.90.67:44423), 74b71844cdbc49cd8087f644f92fa798 (127.7.90.65:43797)
I20260619 11:13:36.951329 26576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a058905995ff4070915bb25faaf819a6" candidate_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0923482d55ec4ce6830b7e3a34a60758"
I20260619 11:13:36.951408 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a058905995ff4070915bb25faaf819a6" candidate_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b71844cdbc49cd8087f644f92fa798"
I20260619 11:13:36.951488 26576 raft_consensus.cc:3037] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:36.951536 26314 raft_consensus.cc:3037] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:36.952648 26576 raft_consensus.cc:2454] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e4363b3e9e47fc85a3ba55e19c1ed9 in term 1.
I20260619 11:13:36.952656 26314 raft_consensus.cc:2454] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e4363b3e9e47fc85a3ba55e19c1ed9 in term 1.
I20260619 11:13:36.952952 26379 leader_election.cc:302] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0923482d55ec4ce6830b7e3a34a60758, 94e4363b3e9e47fc85a3ba55e19c1ed9; no voters:
I20260619 11:13:36.953130 26646 raft_consensus.cc:2781] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:36.953339 26646 raft_consensus.cc:686] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 94e4363b3e9e47fc85a3ba55e19c1ed9, State: Running, Role: LEADER
I20260619 11:13:36.953715 26646 consensus_queue.cc:237] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } }
I20260619 11:13:36.956717 26168 catalog_manager.cc:5495] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94e4363b3e9e47fc85a3ba55e19c1ed9 (127.7.90.66). New cstate: current_term: 1 leader_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94e4363b3e9e47fc85a3ba55e19c1ed9" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 45913 } health_report { overall_health: HEALTHY } } }
I20260619 11:13:36.999126 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:36.995968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.7.90.66:45913)
0619 11:13:36.996095 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:36.996098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:36.996111 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:13:36.996198 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:36.996200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:36.996254 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:36.996337 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:36.996540 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.997021 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.997843 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:36.998048 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:36.998121 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:36.998387 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:36.998769 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:36.998874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:36.998971 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 11:13:36.999002 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:13:37.004509 26314 raft_consensus.cc:1261] T a058905995ff4070915bb25faaf819a6 P 74b71844cdbc49cd8087f644f92fa798 [term 1 FOLLOWER]: Refusing update from remote peer 94e4363b3e9e47fc85a3ba55e19c1ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:37.004508 26576 raft_consensus.cc:1261] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 1 FOLLOWER]: Refusing update from remote peer 94e4363b3e9e47fc85a3ba55e19c1ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:37.005214 26646 consensus_queue.cc:1035] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0923482d55ec4ce6830b7e3a34a60758" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:37.005448 26649 consensus_queue.cc:1035] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b71844cdbc49cd8087f644f92fa798" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:37.013242 26653 mvcc.cc:205] Tried to move back new op lower bound from 7298529759243898880 to 7298529759045087232. Current Snapshot: MvccSnapshot[applied={T|T < 7298529759243898880}]
I20260619 11:13:37.014488 26654 mvcc.cc:205] Tried to move back new op lower bound from 7298529759243898880 to 7298529759045087232. Current Snapshot: MvccSnapshot[applied={T|T < 7298529759243898880}]
I20260619 11:13:37.021322 26171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:37.021508 26171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:37.024037 26171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2325a7fdf9ee453d8ce04d14dfd53da1: Sending DeleteTablet for 3 replicas of tablet a058905995ff4070915bb25faaf819a6
I20260619 11:13:37.024824 26556 tablet_service.cc:1514] Processing DeleteTablet for tablet a058905995ff4070915bb25faaf819a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:37 UTC) from {username='slave'} at 127.0.0.1:53592
I20260619 11:13:37.024868 26425 tablet_service.cc:1514] Processing DeleteTablet for tablet a058905995ff4070915bb25faaf819a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:37 UTC) from {username='slave'} at 127.0.0.1:33084
I20260619 11:13:37.024950 26294 tablet_service.cc:1514] Processing DeleteTablet for tablet a058905995ff4070915bb25faaf819a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:37 UTC) from {username='slave'} at 127.0.0.1:50116
I20260619 11:13:37.025496 26662 tablet_replica.cc:330] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9: stopping tablet replica
I20260619 11:13:37.025502 26661 tablet_replica.cc:330] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: stopping tablet replica
I20260619 11:13:37.025619 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26232
I20260619 11:13:37.026288 26662 raft_consensus.cc:2229] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:13:37.026935 26661 raft_consensus.cc:2229] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:37.027143 26662 raft_consensus.cc:2258] T a058905995ff4070915bb25faaf819a6 P 94e4363b3e9e47fc85a3ba55e19c1ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:37.027243 26661 raft_consensus.cc:2258] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:37.028712 26661 ts_tablet_manager.cc:1905] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:37.030855 26661 ts_tablet_manager.cc:1918] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:37.030965 26661 log.cc:1196] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/a058905995ff4070915bb25faaf819a6
I20260619 11:13:37.031319 26661 ts_tablet_manager.cc:1939] T a058905995ff4070915bb25faaf819a6 P 0923482d55ec4ce6830b7e3a34a60758: Deleting consensus metadata
I20260619 11:13:37.032462 26157 catalog_manager.cc:4841] TS 0923482d55ec4ce6830b7e3a34a60758 (127.7.90.67:44423): tablet a058905995ff4070915bb25faaf819a6 (table test-table [id=d7692a3062584231ac487336f50d5eaf]) successfully deleted
I20260619 11:13:37.034035 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26363
W20260619 11:13:37.034051 26159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:43797 (error 108)
I20260619 11:13:37.035290 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:37.035022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43797 (local address 127.0.0.1:50132)
0619 11:13:37.035144 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:37.035219 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:13:37.035420 26159 catalog_manager.cc:4584] TS 74b71844cdbc49cd8087f644f92fa798 (127.7.90.65:43797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a058905995ff4070915bb25faaf819a6: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43797: connect: Connection refused (error 111)
I20260619 11:13:37.040289 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26494
I20260619 11:13:37.042068 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:37.040358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:45913 (local address 127.0.0.1:33108)
0619 11:13:37.041907 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:37.041973 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.66:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1439}
I20260619 11:13:37.045640 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26141
2026-06-19T11:13:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:37 dist-test-slave-f1t5 krb5kdc[26667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:37 dist-test-slave-f1t5 krb5kdc[26667](info): set up 2 sockets
Jun 19 11:13:37 dist-test-slave-f1t5 krb5kdc[26667](info): commencing operation
krb5kdc: starting...
W20260619 11:13:38.081480 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:13:38.192206 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35259
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:38.298786 26680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:38.299088 26680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:38.299162 26680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:38.303099 26680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:38.303194 26680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:38.303231 26680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:38.303270 26680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:38.303308 26680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:38.308542 26680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:38.309589 26680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:38.310583 26680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:38.318958 26686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.318933 26685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.319064 26680 server_base.cc:1034] running on GCE node
W20260619 11:13:38.318945 26688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.319703 26680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:38.320672 26680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:38.321825 26680 hybrid_clock.cc:648] HybridClock initialized: now 1781867618321813 us; error 24 us; skew 500 ppm
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:38.324519 26680 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:38.325381 26680 webserver.cc:460] Webserver started at http://127.7.90.126:37067/ using document root <none> and password file <none>
I20260619 11:13:38.325942 26680 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:38.326005 26680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:38.326231 26680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:38.328310 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "d0d84c578b434b518d14f1c6c3178228"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "bbe763fd9b04eca3af21416aceefae55"
server_key_iv: "8ad31c24ebd16f1325811fe68360ef02"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.328860 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "d0d84c578b434b518d14f1c6c3178228"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "bbe763fd9b04eca3af21416aceefae55"
server_key_iv: "8ad31c24ebd16f1325811fe68360ef02"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.332350 26680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:13:38.334728 26695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.335858 26680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:13:38.335991 26680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "d0d84c578b434b518d14f1c6c3178228"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "bbe763fd9b04eca3af21416aceefae55"
server_key_iv: "8ad31c24ebd16f1325811fe68360ef02"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.336098 26680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:38.354806 26680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:38.357657 26680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:38.357851 26680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:38.364534 26680 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:35259
I20260619 11:13:38.364562 26747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:35259 every 8 connection(s)
I20260619 11:13:38.365438 26680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:38.367532 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26680
I20260619 11:13:38.367666 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:38.367947 7529 external_mini_cluster.cc:1428] Setting key 91cd49d7b12ec689850b6b40e4c5847f
I20260619 11:13:38.368198 26748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:38.374204 26748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: Bootstrap starting.
I20260619 11:13:38.376231 26748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: Neither blocks nor log segments found. Creating new log.
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:38.376981 26748 log.cc:824] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:38.378798 26748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: No bootstrap required, opened a new log
I20260619 11:13:38.381610 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:38.369275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.7.90.126:35259)
0619 11:13:38.369716 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:38.369731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:38.369763 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:13:38.370738 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:38.370756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:38.371063 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:38.371405 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:38.372170 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.373134 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.374063 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.374584 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.374754 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:38.377200 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:38.377205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:38.377217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:38.377241 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:38.379002 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.379452 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.379457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.379461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.379561 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.379836 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.379838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.379839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.380174 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:38.380323 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:38.380597 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 11:13:38.380871 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":159,"threads_started":1}
I20260619 11:13:38.382840 26748 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } }
I20260619 11:13:38.383091 26748 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:38.383147 26748 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d84c578b434b518d14f1c6c3178228, State: Initialized, Role: FOLLOWER
I20260619 11:13:38.383569 26748 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } }
I20260619 11:13:38.383721 26748 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:38.383790 26748 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:38.383908 26748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:38.384706 26748 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } }
I20260619 11:13:38.385036 26748 leader_election.cc:302] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d84c578b434b518d14f1c6c3178228; no voters:
I20260619 11:13:38.385320 26748 leader_election.cc:288] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:38.385433 26753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:38.385680 26753 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [term 1 LEADER]: Becoming Leader. State: Replica: d0d84c578b434b518d14f1c6c3178228, State: Running, Role: LEADER
I20260619 11:13:38.386072 26753 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } }
I20260619 11:13:38.386631 26748 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:38.387750 26755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d84c578b434b518d14f1c6c3178228. Latest consensus state: current_term: 1 leader_uuid: "d0d84c578b434b518d14f1c6c3178228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } } }
I20260619 11:13:38.387902 26755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:38.388193 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d84c578b434b518d14f1c6c3178228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d84c578b434b518d14f1c6c3178228" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35259 } } }
I20260619 11:13:38.388288 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:38.388302 26762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:38.391247 26762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:38.396242 26762 catalog_manager.cc:1347] Generated new cluster ID: 9a13502e6ee447189b81b44047ce031b
I20260619 11:13:38.396302 26762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:38.404017 26762 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:38.404789 26762 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:38.417644 26762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: Generated new TSK 0
I20260619 11:13:38.418231 26762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.90.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
Entry for principal kudu/127.7.90.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab.
I20260619 11:13:38.446755 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:38.556768 26774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:38.557026 26774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:38.557071 26774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:38.560935 26774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:38.560988 26774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:38.561045 26774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:38.566200 26774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.65.keytab
--principal=kudu/127.7.90.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:38.567196 26774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:38.568149 26774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:38.577430 26782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.577456 26780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.577432 26779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.577636 26774 server_base.cc:1034] running on GCE node
I20260619 11:13:38.578096 26774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:38.578670 26774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:38.579854 26774 hybrid_clock.cc:648] HybridClock initialized: now 1781867618579839 us; error 26 us; skew 500 ppm
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:38.582614 26774 init.cc:377] Logged in from keytab as kudu/127.7.90.65@KRBTEST.COM (short username kudu)
I20260619 11:13:38.583370 26774 webserver.cc:460] Webserver started at http://127.7.90.65:37165/ using document root <none> and password file <none>
I20260619 11:13:38.583868 26774 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:38.583925 26774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:38.584082 26774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:38.586026 26774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "9b7abfa69c634d20b4a2a55aa7aac90d"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "5ede75b6eb29c93c02f57e49fc884096"
server_key_iv: "c8dec4f9d12f2ca80fcf6fe7c52db053"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.586519 26774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "9b7abfa69c634d20b4a2a55aa7aac90d"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "5ede75b6eb29c93c02f57e49fc884096"
server_key_iv: "c8dec4f9d12f2ca80fcf6fe7c52db053"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.590178 26774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:38.592346 26789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.593449 26774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:38.593614 26774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "9b7abfa69c634d20b4a2a55aa7aac90d"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "5ede75b6eb29c93c02f57e49fc884096"
server_key_iv: "c8dec4f9d12f2ca80fcf6fe7c52db053"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.593708 26774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:38.610368 26774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:38.613155 26774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:38.613340 26774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:38.613947 26774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:38.614717 26774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:38.614761 26774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.614822 26774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:38.614852 26774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.623569 26774 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:33353
I20260619 11:13:38.623593 26902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:33353 every 8 connection(s)
I20260619 11:13:38.624405 26774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:38.632755 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26774
I20260619 11:13:38.632905 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.65@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:38.633205 7529 external_mini_cluster.cc:1428] Setting key 74f45f9cc103e31628df5463d6a26abc
I20260619 11:13:38.637717 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:38.626193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:50873 (local address 127.7.90.126:35259)
0619 11:13:38.626312 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:38.626316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:38.627268 (+ 952us) server_negotiation.cc:407] Connection header received
0619 11:13:38.628094 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:38.628097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:38.628143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:38.628216 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:38.629288 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.629898 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.630962 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.631173 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.631248 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:38.633866 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:38.633871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:38.633874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:38.633904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:38.635515 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.636137 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.636140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.636141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.636214 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.636641 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.636645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.636649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.636888 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:38.636997 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:38.637445 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 11:13:38.637603 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:13:38.638479 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:38.626630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35259 (local address 127.7.90.65:50873)
0619 11:13:38.627118 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:38.627159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:38.627905 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:38.628370 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:38.628385 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:38.628715 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:38.629164 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:38.629177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.630003 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.630005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:38.630859 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:38.630868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.631294 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.631296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:38.631443 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:38.631973 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:38.631988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:38.633625 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:38.635634 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:38.635642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:38.635659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:38.636034 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:38.636364 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:38.636370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:38.636373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:38.636548 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:38.636993 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:38.636998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:38.637216 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 11:13:38.637363 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 11:13:38.637707 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260619 11:13:38.639832 26903 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35259
I20260619 11:13:38.640074 26903 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:38.640651 26903 heartbeater.cc:511] Master 127.7.90.126:35259 requested a full tablet report, sending...
I20260619 11:13:38.642382 26712 ts_manager.cc:194] Registered new tserver with Master: 9b7abfa69c634d20b4a2a55aa7aac90d (127.7.90.65:33353)
WARNING: no policy specified for kudu/127.7.90.66@KRBTEST.COM; defaulting to no policy
I20260619 11:13:38.644079 26712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.65@KRBTEST.COM'} at 127.7.90.65:50873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
Entry for principal kudu/127.7.90.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab.
I20260619 11:13:38.660401 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:38.768831 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:38.769073 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:38.769112 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:38.773079 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:38.773135 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:38.773190 26908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:38.778461 26908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.66.keytab
--principal=kudu/127.7.90.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:38.779482 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:38.780443 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:38.789916 26913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.789958 26908 server_base.cc:1034] running on GCE node
W20260619 11:13:38.789902 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.789923 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.790495 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:38.791074 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:38.792263 26908 hybrid_clock.cc:648] HybridClock initialized: now 1781867618792251 us; error 31 us; skew 500 ppm
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:38.794952 26908 init.cc:377] Logged in from keytab as kudu/127.7.90.66@KRBTEST.COM (short username kudu)
I20260619 11:13:38.795797 26908 webserver.cc:460] Webserver started at http://127.7.90.66:41005/ using document root <none> and password file <none>
I20260619 11:13:38.796401 26908 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:38.796489 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:38.796729 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:38.798734 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "b9fc26d6a67d46b5bff70c27447405a6"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "4012e0149c4a9fc15512ff540a971813"
server_key_iv: "d2a94f0c74cefa23c8a0e2bee4409078"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.799288 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "b9fc26d6a67d46b5bff70c27447405a6"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "4012e0149c4a9fc15512ff540a971813"
server_key_iv: "d2a94f0c74cefa23c8a0e2bee4409078"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.807008 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260619 11:13:38.809232 26923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.810539 26908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:38.810688 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "b9fc26d6a67d46b5bff70c27447405a6"
format_stamp: "Formatted at 2026-06-19 11:13:38 on dist-test-slave-f1t5"
server_key: "4012e0149c4a9fc15512ff540a971813"
server_key_iv: "d2a94f0c74cefa23c8a0e2bee4409078"
server_key_version: "encryptionkey@0"
I20260619 11:13:38.810827 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:38.825418 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:38.828981 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:38.829197 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:38.829933 26908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:38.830801 26908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:38.830876 26908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.830947 26908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:38.831002 26908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:38.840942 26908 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:46155
I20260619 11:13:38.840958 27036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:46155 every 8 connection(s)
I20260619 11:13:38.841888 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:38.846257 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 26908
I20260619 11:13:38.846355 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:38.846628 7529 external_mini_cluster.cc:1428] Setting key 6a38ca3eb660b5eb7f38d57e20bd3239
Jun 19 11:13:38 dist-test-slave-f1t5 krb5kdc[26667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867618, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.66@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.90.67@KRBTEST.COM; defaulting to no policy
I20260619 11:13:38.855983 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:38.843680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:54745 (local address 127.7.90.126:35259)
0619 11:13:38.843845 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:38.843849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:38.844857 (+ 1008us) server_negotiation.cc:407] Connection header received
0619 11:13:38.845816 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:38.845819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:38.845880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:38.845966 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:38.847317 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.847870 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.849059 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.849315 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.849438 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:38.852388 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:38.852392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:38.852395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:38.852426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:38.853944 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.854448 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.854453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.854455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.854535 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:38.854884 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:38.854889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:38.854891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:38.855070 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:38.855175 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:38.855707 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 11:13:38.855823 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 11:13:38.856848 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:38.844047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35259 (local address 127.7.90.66:54745)
0619 11:13:38.844681 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:38.844730 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:38.845582 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:38.846145 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:38.846152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:38.846613 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:38.847151 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:38.847167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.848003 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.848007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:38.848919 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:38.848929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:38.849472 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:38.849476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:38.849704 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:38.850341 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:38.850358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:38.852199 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:38.854076 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:38.854081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:38.854095 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:38.854347 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:38.854643 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:38.854647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:38.854649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:38.854784 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:38.855174 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:38.855180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:38.855480 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 11:13:38.855664 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:13:38.855980 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.67@KRBTEST.COM" created.
I20260619 11:13:38.858142 27037 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35259
I20260619 11:13:38.858397 27037 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:38.858989 27037 heartbeater.cc:511] Master 127.7.90.126:35259 requested a full tablet report, sending...
I20260619 11:13:38.860189 26712 ts_manager.cc:194] Registered new tserver with Master: b9fc26d6a67d46b5bff70c27447405a6 (127.7.90.66:46155)
I20260619 11:13:38.860842 26712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.66@KRBTEST.COM'} at 127.7.90.66:54745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
Entry for principal kudu/127.7.90.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab.
I20260619 11:13:38.872843 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:38.977183 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:38.977401 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:38.977473 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:38.981320 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:38.981369 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:38.981441 27042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:38.986513 27042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.67.keytab
--principal=kudu/127.7.90.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:38.987450 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:38.988371 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:38.997366 27048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.997357 27047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:38.997380 27050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:38.997905 27042 server_base.cc:1034] running on GCE node
I20260619 11:13:38.998256 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:38.998808 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:38.999989 27042 hybrid_clock.cc:648] HybridClock initialized: now 1781867618999967 us; error 30 us; skew 500 ppm
Jun 19 11:13:39 dist-test-slave-f1t5 krb5kdc[26667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:39.002610 27042 init.cc:377] Logged in from keytab as kudu/127.7.90.67@KRBTEST.COM (short username kudu)
I20260619 11:13:39.003428 27042 webserver.cc:460] Webserver started at http://127.7.90.67:42351/ using document root <none> and password file <none>
I20260619 11:13:39.003953 27042 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:39.004038 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:39.004251 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:39.006189 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "fbe7715ffccf464c91e23bb32dc72bec"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "a77ffa9e1cae6744eeae74b64c54b93c"
server_key_iv: "377706f27f6be79b6fdbc533213ecfcd"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.006714 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "fbe7715ffccf464c91e23bb32dc72bec"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "a77ffa9e1cae6744eeae74b64c54b93c"
server_key_iv: "377706f27f6be79b6fdbc533213ecfcd"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.010051 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:13:39.012204 27057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.013290 27042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:39.013437 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "fbe7715ffccf464c91e23bb32dc72bec"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "a77ffa9e1cae6744eeae74b64c54b93c"
server_key_iv: "377706f27f6be79b6fdbc533213ecfcd"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.013586 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:39.033684 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:39.036284 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:39.036475 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:39.037005 27042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:39.037797 27042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:39.037873 27042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.037937 27042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:39.037992 27042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.046303 27042 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:44627
I20260619 11:13:39.046330 27170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:44627 every 8 connection(s)
I20260619 11:13:39.047092 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:39.048588 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27042
I20260619 11:13:39.048698 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:39.049021 7529 external_mini_cluster.cc:1428] Setting key 8d55d0b436844d6ec4845e9c667e9316
Jun 19 11:13:39 dist-test-slave-f1t5 krb5kdc[26667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.67@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:39.059545 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.048739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:46517 (local address 127.7.90.126:35259)
0619 11:13:39.048902 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.048906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.049771 (+ 865us) server_negotiation.cc:407] Connection header received
0619 11:13:39.050888 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.050891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.050942 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.051021 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:39.052218 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.052759 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.053944 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.054163 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.054272 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.056503 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:39.056506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:39.056508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:39.056531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:39.057794 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:39.058208 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:39.058210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:39.058211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:39.058280 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:39.058539 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:39.058540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:39.058542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:39.058732 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:39.058812 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.059327 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.059430 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:13:39.060227 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.049076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35259 (local address 127.7.90.67:46517)
0619 11:13:39.049620 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.049670 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.050689 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.051185 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.051194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.051630 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:39.052088 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.052100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.052879 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.052883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.053828 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.053838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.054303 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.054307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.054480 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.055033 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:13:39.055048 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:39.056366 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:39.057888 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:39.057892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:39.057902 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:39.058129 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:39.058350 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:13:39.058351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:13:39.058353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:13:39.058467 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:13:39.058820 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:39.058824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:39.059014 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.059153 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.059399 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260619 11:13:39.061311 27171 heartbeater.cc:347] Connected to a master server at 127.7.90.126:35259
I20260619 11:13:39.061601 27171 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:39.062158 27171 heartbeater.cc:511] Master 127.7.90.126:35259 requested a full tablet report, sending...
I20260619 11:13:39.063169 26712 ts_manager.cc:194] Registered new tserver with Master: fbe7715ffccf464c91e23bb32dc72bec (127.7.90.67:44627)
I20260619 11:13:39.063719 26712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.90.67@KRBTEST.COM'} at 127.7.90.67:46517
I20260619 11:13:39.074721 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:39.083556 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.076497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.7.90.126:35259)
0619 11:13:39.076628 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.076634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.076791 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:13:39.076956 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.076959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.077004 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.077087 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:39.077779 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.078275 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.079084 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.079288 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.079388 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.080662 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:39.080666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:39.080669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:39.080715 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:39.081972 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:39.082354 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:39.082359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:39.082360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:39.082436 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:39.082713 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:39.082718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:39.082719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:39.082924 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:39.083063 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.083291 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.083411 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:13:39.086153 26712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:39.089242 26712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:39.102669 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.097000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44627 (local address 127.0.0.1:58044)
0619 11:13:39.097344 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.097389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.097482 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.098342 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.098348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.098384 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:39.098516 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.098523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.099939 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.099945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.101026 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.101035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.102068 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.102072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.102258 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.102298 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.102422 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.102539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":22,"thread_start_us":103,"threads_started":1}
I20260619 11:13:39.102736 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.097138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.7.90.67:44627)
0619 11:13:39.097819 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.097833 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.097856 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:13:39.097926 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.097931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.098129 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.098344 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:39.098867 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.099771 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.101156 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.101935 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.102068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.102208 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.102469 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.102568 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":130,"threads_started":1}
I20260619 11:13:39.103286 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.097762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:46155 (local address 127.0.0.1:40988)
0619 11:13:39.097967 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.097982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.098117 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.098947 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.098950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.098966 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:39.099049 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.099055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.100496 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.100499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.101584 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.101594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.102889 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.102893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.102983 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.103011 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.103098 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.103166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260619 11:13:39.103403 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.097926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.7.90.66:46155)
0619 11:13:39.098357 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.098369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.098390 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:13:39.098463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.098468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.098645 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.098792 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:39.099183 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.100343 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.101696 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.102577 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.102676 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.102804 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.103139 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.103228 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":102,"threads_started":1}
I20260619 11:13:39.103852 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.098093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:33353 (local address 127.0.0.1:53120)
0619 11:13:39.099070 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.099080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.099194 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.099684 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.099687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.099704 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:39.099772 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.099778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.100983 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.100989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.102241 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.102252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.103487 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.103491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.103574 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.103594 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.103669 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.103730 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260619 11:13:39.103931 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.098595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.7.90.65:33353)
0619 11:13:39.098960 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.098972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.099288 (+ 316us) server_negotiation.cc:407] Connection header received
0619 11:13:39.099362 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.099367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.099534 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.099690 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:39.099911 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.100831 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.102379 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.103179 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.103279 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.103426 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.103712 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.103786 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260619 11:13:39.105990 27105 tablet_service.cc:1467] Processing CreateTablet for tablet eae13a7c7205402faa1362c168fcb2ea (DEFAULT_TABLE table=test-table [id=10bbdac4560d42fcbd9b5cee04de3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:39.106926 27105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae13a7c7205402faa1362c168fcb2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:39.106644 26837 tablet_service.cc:1467] Processing CreateTablet for tablet eae13a7c7205402faa1362c168fcb2ea (DEFAULT_TABLE table=test-table [id=10bbdac4560d42fcbd9b5cee04de3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:39.106637 26971 tablet_service.cc:1467] Processing CreateTablet for tablet eae13a7c7205402faa1362c168fcb2ea (DEFAULT_TABLE table=test-table [id=10bbdac4560d42fcbd9b5cee04de3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:39.107601 26837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae13a7c7205402faa1362c168fcb2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:39.107601 26971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae13a7c7205402faa1362c168fcb2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:39.113035 27188 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Bootstrap starting.
I20260619 11:13:39.115267 27189 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Bootstrap starting.
I20260619 11:13:39.115684 27188 tablet_bootstrap.cc:654] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:39.116039 27190 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: Bootstrap starting.
I20260619 11:13:39.116750 27188 log.cc:824] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:39.118202 27190 tablet_bootstrap.cc:654] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:39.118845 27188 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: No bootstrap required, opened a new log
I20260619 11:13:39.118942 27189 tablet_bootstrap.cc:654] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:39.119093 27188 ts_tablet_manager.cc:1397] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 11:13:39.119426 27190 log.cc:824] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:39.119673 27189 log.cc:824] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:39.121166 27190 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: No bootstrap required, opened a new log
I20260619 11:13:39.121353 27190 ts_tablet_manager.cc:1397] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:13:39.121613 27189 tablet_bootstrap.cc:492] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: No bootstrap required, opened a new log
I20260619 11:13:39.121773 27189 ts_tablet_manager.cc:1397] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:13:39.122967 27188 raft_consensus.cc:348] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.123224 27188 raft_consensus.cc:374] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:39.123307 27188 raft_consensus.cc:729] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe7715ffccf464c91e23bb32dc72bec, State: Initialized, Role: FOLLOWER
I20260619 11:13:39.123750 27188 consensus_queue.cc:260] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.124397 27190 raft_consensus.cc:348] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.124698 27188 ts_tablet_manager.cc:1428] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 11:13:39.124706 27190 raft_consensus.cc:374] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:39.124758 27190 raft_consensus.cc:729] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7abfa69c634d20b4a2a55aa7aac90d, State: Initialized, Role: FOLLOWER
I20260619 11:13:39.125065 27171 heartbeater.cc:503] Master 127.7.90.126:35259 was elected leader, sending a full tablet report...
I20260619 11:13:39.125243 27190 consensus_queue.cc:260] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.125255 27189 raft_consensus.cc:348] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.125489 27189 raft_consensus.cc:374] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:39.125607 27189 raft_consensus.cc:729] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fc26d6a67d46b5bff70c27447405a6, State: Initialized, Role: FOLLOWER
I20260619 11:13:39.126025 27189 consensus_queue.cc:260] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.126214 26903 heartbeater.cc:503] Master 127.7.90.126:35259 was elected leader, sending a full tablet report...
I20260619 11:13:39.126212 27190 ts_tablet_manager.cc:1428] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:13:39.126748 27037 heartbeater.cc:503] Master 127.7.90.126:35259 was elected leader, sending a full tablet report...
I20260619 11:13:39.126946 27189 ts_tablet_manager.cc:1428] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:13:39.197954 27195 raft_consensus.cc:482] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:39.198127 27195 raft_consensus.cc:504] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.199111 27195 leader_election.cc:288] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe7715ffccf464c91e23bb32dc72bec (127.7.90.67:44627), b9fc26d6a67d46b5bff70c27447405a6 (127.7.90.66:46155)
I20260619 11:13:39.204138 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.199846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:40835 (local address 127.7.90.66:46155)
0619 11:13:39.199967 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.199971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.200107 (+ 136us) server_negotiation.cc:407] Connection header received
0619 11:13:39.200182 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.200184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.200246 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.200322 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:39.200557 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.201264 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.202812 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.203430 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.203532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.203593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.203945 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.204013 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:13:39.204138 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.199564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:34471 (local address 127.7.90.67:44627)
0619 11:13:39.199701 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.199706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.199723 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:13:39.199777 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.199782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.199835 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.199928 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:39.200283 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.201094 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.202812 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.203423 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.203532 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.203593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.203945 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.204013 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:13:39.204252 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.199682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:46155 (local address 127.7.90.65:40835)
0619 11:13:39.200009 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.200021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.200107 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.200342 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.200344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.200357 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:39.200448 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.200452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.201350 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.201353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.202689 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.202699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.203542 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.203546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.203740 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.203836 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.203915 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.204069 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":191,"thread_start_us":144,"threads_started":1}
I20260619 11:13:39.204258 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.199321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:44627 (local address 127.7.90.65:34471)
0619 11:13:39.199557 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.199574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.199665 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.200005 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.200008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.200031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:39.200164 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.200171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.201194 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.201196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.202689 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.202699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.203542 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.203546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.203704 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.203836 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.203915 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.204069 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":188,"thread_start_us":98,"threads_started":1}
I20260619 11:13:39.204895 26991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae13a7c7205402faa1362c168fcb2ea" candidate_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fc26d6a67d46b5bff70c27447405a6" is_pre_election: true
I20260619 11:13:39.205103 27125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae13a7c7205402faa1362c168fcb2ea" candidate_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe7715ffccf464c91e23bb32dc72bec" is_pre_election: true
I20260619 11:13:39.205181 26991 raft_consensus.cc:2454] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b7abfa69c634d20b4a2a55aa7aac90d in term 0.
I20260619 11:13:39.205438 27125 raft_consensus.cc:2454] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b7abfa69c634d20b4a2a55aa7aac90d in term 0.
I20260619 11:13:39.205685 26793 leader_election.cc:302] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b7abfa69c634d20b4a2a55aa7aac90d, b9fc26d6a67d46b5bff70c27447405a6; no voters:
I20260619 11:13:39.205902 27195 raft_consensus.cc:2781] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:39.205974 27195 raft_consensus.cc:482] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:39.206003 27195 raft_consensus.cc:3037] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:39.207010 27195 raft_consensus.cc:504] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.207347 27195 leader_election.cc:288] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [CANDIDATE]: Term 1 election: Requested vote from peers fbe7715ffccf464c91e23bb32dc72bec (127.7.90.67:44627), b9fc26d6a67d46b5bff70c27447405a6 (127.7.90.66:46155)
I20260619 11:13:39.207666 27125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae13a7c7205402faa1362c168fcb2ea" candidate_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe7715ffccf464c91e23bb32dc72bec"
I20260619 11:13:39.207787 26991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae13a7c7205402faa1362c168fcb2ea" candidate_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9fc26d6a67d46b5bff70c27447405a6"
I20260619 11:13:39.207772 27125 raft_consensus.cc:3037] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:39.207928 26991 raft_consensus.cc:3037] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:39.208885 27125 raft_consensus.cc:2454] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b7abfa69c634d20b4a2a55aa7aac90d in term 1.
I20260619 11:13:39.209022 26991 raft_consensus.cc:2454] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b7abfa69c634d20b4a2a55aa7aac90d in term 1.
I20260619 11:13:39.209167 26793 leader_election.cc:302] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b7abfa69c634d20b4a2a55aa7aac90d, fbe7715ffccf464c91e23bb32dc72bec; no voters:
I20260619 11:13:39.209360 27195 raft_consensus.cc:2781] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:39.209623 27195 raft_consensus.cc:686] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [term 1 LEADER]: Becoming Leader. State: Replica: 9b7abfa69c634d20b4a2a55aa7aac90d, State: Running, Role: LEADER
I20260619 11:13:39.210016 27195 consensus_queue.cc:237] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } }
I20260619 11:13:39.213158 26711 catalog_manager.cc:5495] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b7abfa69c634d20b4a2a55aa7aac90d (127.7.90.65). New cstate: current_term: 1 leader_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b7abfa69c634d20b4a2a55aa7aac90d" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 33353 } health_report { overall_health: HEALTHY } } }
I20260619 11:13:39.228628 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.225071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.7.90.65:33353)
0619 11:13:39.225218 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.225224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.225266 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:13:39.225465 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.225468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.225563 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.225676 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:39.225924 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.226417 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.227258 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.227477 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.227583 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.227891 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:39.228274 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:39.228404 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.228444 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.228493 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 11:13:39.234220 27125 raft_consensus.cc:1261] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 1 FOLLOWER]: Refusing update from remote peer 9b7abfa69c634d20b4a2a55aa7aac90d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:39.234220 26991 raft_consensus.cc:1261] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 1 FOLLOWER]: Refusing update from remote peer 9b7abfa69c634d20b4a2a55aa7aac90d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:39.234890 27195 consensus_queue.cc:1035] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9fc26d6a67d46b5bff70c27447405a6" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:39.235106 27199 consensus_queue.cc:1035] T eae13a7c7205402faa1362c168fcb2ea P 9b7abfa69c634d20b4a2a55aa7aac90d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe7715ffccf464c91e23bb32dc72bec" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:39.243012 27202 mvcc.cc:205] Tried to move back new op lower bound from 7298529768375885824 to 7298529768286851072. Current Snapshot: MvccSnapshot[applied={T|T < 7298529768375885824}]
I20260619 11:13:39.245290 27204 mvcc.cc:205] Tried to move back new op lower bound from 7298529768375885824 to 7298529768286851072. Current Snapshot: MvccSnapshot[applied={T|T < 7298529768375885824}]
I20260619 11:13:39.247156 27203 mvcc.cc:205] Tried to move back new op lower bound from 7298529768375885824 to 7298529768286851072. Current Snapshot: MvccSnapshot[applied={T|T < 7298529768375885824 or (T in {7298529768375885824})}]
I20260619 11:13:39.250595 26711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:39.250757 26711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:39.253156 26711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0d84c578b434b518d14f1c6c3178228: Sending DeleteTablet for 3 replicas of tablet eae13a7c7205402faa1362c168fcb2ea
I20260619 11:13:39.253882 27105 tablet_service.cc:1514] Processing DeleteTablet for tablet eae13a7c7205402faa1362c168fcb2ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:39 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:58044
I20260619 11:13:39.253931 26971 tablet_service.cc:1514] Processing DeleteTablet for tablet eae13a7c7205402faa1362c168fcb2ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:39 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:40988
I20260619 11:13:39.254053 26837 tablet_service.cc:1514] Processing DeleteTablet for tablet eae13a7c7205402faa1362c168fcb2ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:39 UTC) from {username='kudu', principal='kudu/127.7.90.126@KRBTEST.COM'} at 127.0.0.1:53120
I20260619 11:13:39.254325 27213 tablet_replica.cc:330] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: stopping tablet replica
I20260619 11:13:39.254488 27212 tablet_replica.cc:330] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: stopping tablet replica
I20260619 11:13:39.254691 27213 raft_consensus.cc:2229] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:39.254807 27212 raft_consensus.cc:2229] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:39.255041 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26774
I20260619 11:13:39.255048 27213 raft_consensus.cc:2258] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:39.255116 27212 raft_consensus.cc:2258] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:39.257226 27213 ts_tablet_manager.cc:1905] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:39.257333 27212 ts_tablet_manager.cc:1905] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:39.259455 27213 ts_tablet_manager.cc:1918] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:39.259572 27213 log.cc:1196] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/eae13a7c7205402faa1362c168fcb2ea
I20260619 11:13:39.259938 27213 ts_tablet_manager.cc:1939] T eae13a7c7205402faa1362c168fcb2ea P fbe7715ffccf464c91e23bb32dc72bec: Deleting consensus metadata
I20260619 11:13:39.260659 27212 ts_tablet_manager.cc:1918] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:39.260756 27212 log.cc:1196] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/eae13a7c7205402faa1362c168fcb2ea
I20260619 11:13:39.260928 26699 catalog_manager.cc:4841] TS fbe7715ffccf464c91e23bb32dc72bec (127.7.90.67:44627): tablet eae13a7c7205402faa1362c168fcb2ea (table test-table [id=10bbdac4560d42fcbd9b5cee04de3943]) successfully deleted
I20260619 11:13:39.261114 27212 ts_tablet_manager.cc:1939] T eae13a7c7205402faa1362c168fcb2ea P b9fc26d6a67d46b5bff70c27447405a6: Deleting consensus metadata
I20260619 11:13:39.262238 26699 catalog_manager.cc:4841] TS b9fc26d6a67d46b5bff70c27447405a6 (127.7.90.66:46155): tablet eae13a7c7205402faa1362c168fcb2ea (table test-table [id=10bbdac4560d42fcbd9b5cee04de3943]) successfully deleted
W20260619 11:13:39.262990 26699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:33353 (error 108)
I20260619 11:13:39.263161 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26908
I20260619 11:13:39.264377 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.263816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:33353 (local address 127.0.0.1:53146)
0619 11:13:39.264244 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.264311 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
W20260619 11:13:39.264555 26699 catalog_manager.cc:4584] TS 9b7abfa69c634d20b4a2a55aa7aac90d (127.7.90.65:33353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eae13a7c7205402faa1362c168fcb2ea: Network error: Client connection negotiation failed: client connection to 127.7.90.65:33353: connect: Connection refused (error 111)
I20260619 11:13:39.268911 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27042
I20260619 11:13:39.273928 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 26680
2026-06-19T11:13:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2228 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T11:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:39Z Disabled control of system clock
I20260619 11:13:39.302992 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41155
--webserver_interface=127.7.90.126
--webserver_port=0
--builtin_ntp_servers=127.7.90.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:39.409101 27219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:39.409313 27219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:39.409345 27219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:39.413247 27219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:39.413295 27219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:39.413309 27219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:39.413324 27219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:39.413380 27219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:39.418004 27219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:41155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:39.418888 27219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:39.420110 27219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:39.427968 27225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:39.427951 27224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:39.428016 27219 server_base.cc:1034] running on GCE node
W20260619 11:13:39.427956 27227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:39.428699 27219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:39.429700 27219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:39.430850 27219 hybrid_clock.cc:648] HybridClock initialized: now 1781867619430824 us; error 38 us; skew 500 ppm
I20260619 11:13:39.432432 27219 webserver.cc:460] Webserver started at http://127.7.90.126:45135/ using document root <none> and password file <none>
I20260619 11:13:39.432963 27219 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:39.433022 27219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:39.433234 27219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:39.435177 27219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "fca24f23ccd04632a4d0085616547d1e"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "d854cfdb68cd792fab5d25fd9db739a6"
server_key_iv: "5a072c0f4b403f3b4540166525b18428"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.435719 27219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "fca24f23ccd04632a4d0085616547d1e"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "d854cfdb68cd792fab5d25fd9db739a6"
server_key_iv: "5a072c0f4b403f3b4540166525b18428"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.439239 27219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:39.441505 27233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.442760 27219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:39.442893 27219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "fca24f23ccd04632a4d0085616547d1e"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "d854cfdb68cd792fab5d25fd9db739a6"
server_key_iv: "5a072c0f4b403f3b4540166525b18428"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.442996 27219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:39.457191 27219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:39.457866 27219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:39.458046 27219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:39.465054 27219 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:41155
I20260619 11:13:39.465071 27285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:41155 every 8 connection(s)
I20260619 11:13:39.466048 27219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:39.467876 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27219
I20260619 11:13:39.468020 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:39.468290 7529 external_mini_cluster.cc:1428] Setting key f27ee5f142e7530581770fd7b79d138c
I20260619 11:13:39.469241 27286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:39.474678 27286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: Bootstrap starting.
I20260619 11:13:39.476907 27286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:39.477109 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.469684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.7.90.126:41155)
0619 11:13:39.470304 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.470317 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.470350 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:13:39.470981 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.470996 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.471347 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.471676 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:39.472080 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.472986 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.473982 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.474447 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.474628 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.475459 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:39.475463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:39.475474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:39.475828 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:39.475881 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:39.475901 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:39.476016 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.476159 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.476402 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260619 11:13:39.477739 27286 log.cc:824] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:39.479465 27286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: No bootstrap required, opened a new log
I20260619 11:13:39.482802 27286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } }
I20260619 11:13:39.483076 27286 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:39.483158 27286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca24f23ccd04632a4d0085616547d1e, State: Initialized, Role: FOLLOWER
I20260619 11:13:39.483603 27286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } }
I20260619 11:13:39.483743 27286 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:39.483806 27286 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:39.483922 27286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:39.484721 27286 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } }
I20260619 11:13:39.485044 27286 leader_election.cc:302] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca24f23ccd04632a4d0085616547d1e; no voters:
I20260619 11:13:39.485328 27286 leader_election.cc:288] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:39.485445 27291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:39.485692 27291 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [term 1 LEADER]: Becoming Leader. State: Replica: fca24f23ccd04632a4d0085616547d1e, State: Running, Role: LEADER
I20260619 11:13:39.486018 27291 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } }
I20260619 11:13:39.486445 27286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:39.487459 27293 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca24f23ccd04632a4d0085616547d1e. Latest consensus state: current_term: 1 leader_uuid: "fca24f23ccd04632a4d0085616547d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } } }
I20260619 11:13:39.487572 27293 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:39.487916 27292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca24f23ccd04632a4d0085616547d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca24f23ccd04632a4d0085616547d1e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41155 } } }
I20260619 11:13:39.488054 27300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:39.488075 27292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:39.491045 27300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:39.496737 27300 catalog_manager.cc:1347] Generated new cluster ID: 7f12c956ef5044a28f5af38f3cf0b3a3
I20260619 11:13:39.496826 27300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:39.524537 27300 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:39.525815 27300 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:39.534821 27300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: Generated new TSK 0
I20260619 11:13:39.535586 27300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:39.542173 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.65:0
--local_ip_for_outbound_sockets=127.7.90.65
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--builtin_ntp_servers=127.7.90.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:13:39.653323 27310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:39.653640 27310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:39.653718 27310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:39.657689 27310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:39.657779 27310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:39.657871 27310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.65
I20260619 11:13:39.662535 27310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.90.65
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.65
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:39.663599 27310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:39.664964 27310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:39.674311 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:39.674319 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:39.674330 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:39.674674 27310 server_base.cc:1034] running on GCE node
I20260619 11:13:39.674999 27310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:39.675594 27310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:39.676753 27310 hybrid_clock.cc:648] HybridClock initialized: now 1781867619676734 us; error 32 us; skew 500 ppm
I20260619 11:13:39.678485 27310 webserver.cc:460] Webserver started at http://127.7.90.65:46489/ using document root <none> and password file <none>
I20260619 11:13:39.678983 27310 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:39.679040 27310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:39.679193 27310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:39.681145 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data/instance:
uuid: "d93d6a729ccd4535bd746f80679052df"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "e9c193f495587fcc1dfeebacc775310f"
server_key_iv: "6045036a84d2be74b042223cd0b35d5d"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.681720 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance:
uuid: "d93d6a729ccd4535bd746f80679052df"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "e9c193f495587fcc1dfeebacc775310f"
server_key_iv: "6045036a84d2be74b042223cd0b35d5d"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.685094 27310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:13:39.687661 27324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.688995 27310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 11:13:39.689088 27310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal
uuid: "d93d6a729ccd4535bd746f80679052df"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "e9c193f495587fcc1dfeebacc775310f"
server_key_iv: "6045036a84d2be74b042223cd0b35d5d"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.689170 27310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:39.702073 27310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:39.702631 27310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:39.702764 27310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:39.703217 27310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:39.704047 27310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:39.704092 27310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.704129 27310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:39.704144 27310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.712809 27310 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.65:43487
I20260619 11:13:39.712885 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.65:43487 every 8 connection(s)
I20260619 11:13:39.713778 27310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/data/info.pb
I20260619 11:13:39.718410 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27310
I20260619 11:13:39.718513 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-0/wal/instance
I20260619 11:13:39.718820 7529 external_mini_cluster.cc:1428] Setting key c3ebb9debf7255e637d4c186ed5f1b25
I20260619 11:13:39.720700 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.66:0
--local_ip_for_outbound_sockets=127.7.90.66
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--builtin_ntp_servers=127.7.90.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:39.722978 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.715469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.65:50153 (local address 127.7.90.126:41155)
0619 11:13:39.715619 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.715623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.716658 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 11:13:39.717980 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.717983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.718033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.718118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:39.719141 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.719792 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.721000 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.721216 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.721328 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.722527 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:39.722530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:39.722532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:39.722543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:39.722561 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:39.722569 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:39.722628 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.722802 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.722863 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:13:39.723857 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.715891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41155 (local address 127.7.90.65:50153)
0619 11:13:39.716534 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.716572 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.717800 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.718273 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.718280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.718357 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:39.718999 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.719015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.719920 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.719923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.720809 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.720823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.721362 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.721366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.721578 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.722230 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:39.722233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:39.722288 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:39.722370 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:39.722654 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:39.722664 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:39.722671 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.722799 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.723117 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
I20260619 11:13:39.725025 27438 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41155
I20260619 11:13:39.725286 27438 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:39.725912 27438 heartbeater.cc:511] Master 127.7.90.126:41155 requested a full tablet report, sending...
I20260619 11:13:39.727469 27250 ts_manager.cc:194] Registered new tserver with Master: d93d6a729ccd4535bd746f80679052df (127.7.90.65:43487)
I20260619 11:13:39.728607 27250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.65:50153
W20260619 11:13:39.828642 27441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:39.828863 27441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:39.828908 27441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:39.832811 27441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:39.832854 27441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:39.832916 27441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.66
I20260619 11:13:39.837461 27441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.90.66
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.66
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:39.838390 27441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:39.839685 27441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:39.849500 27449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:39.849704 27447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:39.849550 27446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:39.850138 27441 server_base.cc:1034] running on GCE node
I20260619 11:13:39.850675 27441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:39.851562 27441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:39.852792 27441 hybrid_clock.cc:648] HybridClock initialized: now 1781867619852775 us; error 47 us; skew 500 ppm
I20260619 11:13:39.854805 27441 webserver.cc:460] Webserver started at http://127.7.90.66:35059/ using document root <none> and password file <none>
I20260619 11:13:39.855369 27441 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:39.855455 27441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:39.855659 27441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:39.857715 27441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data/instance:
uuid: "bf423097bbda43aaa047d426efec7b5c"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "44e7c21516c51d32991532d46f070a92"
server_key_iv: "fa4ec19389accbdebdaaacf069ae594f"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.858271 27441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance:
uuid: "bf423097bbda43aaa047d426efec7b5c"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "44e7c21516c51d32991532d46f070a92"
server_key_iv: "fa4ec19389accbdebdaaacf069ae594f"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.861789 27441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:13:39.864413 27455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.865506 27441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:39.865687 27441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal
uuid: "bf423097bbda43aaa047d426efec7b5c"
format_stamp: "Formatted at 2026-06-19 11:13:39 on dist-test-slave-f1t5"
server_key: "44e7c21516c51d32991532d46f070a92"
server_key_iv: "fa4ec19389accbdebdaaacf069ae594f"
server_key_version: "encryptionkey@0"
I20260619 11:13:39.865801 27441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:39.895129 27441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:39.895850 27441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:39.896070 27441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:39.896674 27441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:39.897639 27441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:39.897711 27441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.897783 27441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:39.897826 27441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:39.906355 27441 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.66:36499
I20260619 11:13:39.906389 27568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.66:36499 every 8 connection(s)
I20260619 11:13:39.907212 27441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/data/info.pb
I20260619 11:13:39.909718 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27441
I20260619 11:13:39.909848 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal/instance
I20260619 11:13:39.910135 7529 external_mini_cluster.cc:1428] Setting key 6ecde83f3cef3718b33f18fe452d20b8
I20260619 11:13:39.912312 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.90.67:0
--local_ip_for_outbound_sockets=127.7.90.67
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--builtin_ntp_servers=127.7.90.84:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:13:39.917143 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.908949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:35433 (local address 127.7.90.126:41155)
0619 11:13:39.909090 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:39.909094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:39.910288 (+ 1194us) server_negotiation.cc:407] Connection header received
0619 11:13:39.911966 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:39.911968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:39.912012 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:39.912077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:39.913225 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.913809 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.915304 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.915505 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.915614 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.916654 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:39.916658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:39.916660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:39.916678 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:39.916696 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:39.916704 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:39.916776 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:39.916957 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 11:13:39.917012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:39.918051 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:39.909328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41155 (local address 127.7.90.66:35433)
0619 11:13:39.910133 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:39.910191 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:39.911731 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:39.912260 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:39.912269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:39.912388 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:39.913073 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.913095 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.913938 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.913943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.915203 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:39.915213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:39.915652 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:39.915657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:39.915811 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:39.916396 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:39.916398 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:39.916441 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:39.916498 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:39.916808 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:39.916817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:39.916823 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:13:39.916953 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 11:13:39.917209 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":169,"threads_started":1}
I20260619 11:13:39.919203 27569 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41155
I20260619 11:13:39.919410 27569 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:39.919890 27569 heartbeater.cc:511] Master 127.7.90.126:41155 requested a full tablet report, sending...
I20260619 11:13:39.920841 27250 ts_manager.cc:194] Registered new tserver with Master: bf423097bbda43aaa047d426efec7b5c (127.7.90.66:36499)
I20260619 11:13:39.921339 27250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.66:35433
W20260619 11:13:40.019745 27572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:40.020018 27572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:40.020103 27572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:40.024071 27572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:40.024149 27572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:40.024237 27572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.90.67
I20260619 11:13:40.028925 27572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.90.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.90.67
--webserver_port=0
--tserver_master_addrs=127.7.90.126:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.90.67
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:40.029984 27572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:40.031270 27572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:40.040501 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:40.040512 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:40.040530 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:40.040714 27572 server_base.cc:1034] running on GCE node
I20260619 11:13:40.041222 27572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:40.041872 27572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:40.043025 27572 hybrid_clock.cc:648] HybridClock initialized: now 1781867620042991 us; error 45 us; skew 500 ppm
I20260619 11:13:40.044763 27572 webserver.cc:460] Webserver started at http://127.7.90.67:45143/ using document root <none> and password file <none>
I20260619 11:13:40.045313 27572 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:40.045403 27572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:40.045687 27572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:40.047835 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data/instance:
uuid: "e844b97fc5564ec994da37169bcdc221"
format_stamp: "Formatted at 2026-06-19 11:13:40 on dist-test-slave-f1t5"
server_key: "f7535e779e60df32756894959035960d"
server_key_iv: "6147898b88eccda0f179d0caa82fd45e"
server_key_version: "encryptionkey@0"
I20260619 11:13:40.048367 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance:
uuid: "e844b97fc5564ec994da37169bcdc221"
format_stamp: "Formatted at 2026-06-19 11:13:40 on dist-test-slave-f1t5"
server_key: "f7535e779e60df32756894959035960d"
server_key_iv: "6147898b88eccda0f179d0caa82fd45e"
server_key_version: "encryptionkey@0"
I20260619 11:13:40.051817 27572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:40.054260 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:40.055428 27572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:40.055576 27572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal
uuid: "e844b97fc5564ec994da37169bcdc221"
format_stamp: "Formatted at 2026-06-19 11:13:40 on dist-test-slave-f1t5"
server_key: "f7535e779e60df32756894959035960d"
server_key_iv: "6147898b88eccda0f179d0caa82fd45e"
server_key_version: "encryptionkey@0"
I20260619 11:13:40.055691 27572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:40.075730 27572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:40.076367 27572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:40.076551 27572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:40.077121 27572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:13:40.078100 27572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:13:40.078171 27572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:40.078243 27572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:13:40.078302 27572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:40.086844 27572 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.67:40645
I20260619 11:13:40.086884 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.67:40645 every 8 connection(s)
I20260619 11:13:40.087824 27572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/data/info.pb
I20260619 11:13:40.089000 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27572
I20260619 11:13:40.089112 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal/instance
I20260619 11:13:40.089416 7529 external_mini_cluster.cc:1428] Setting key dd79745db44af5185f42bebfba1fbc27
I20260619 11:13:40.097035 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.089865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.67:55193 (local address 127.7.90.126:41155)
0619 11:13:40.090003 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.090007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.091292 (+ 1285us) server_negotiation.cc:407] Connection header received
0619 11:13:40.092767 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.092769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.092832 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.092899 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:40.093816 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.094317 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.095440 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.095638 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.095712 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.096656 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:40.096659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:40.096661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:40.096671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:40.096689 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:40.096696 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:40.096752 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.096887 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.096930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:13:40.097930 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.090393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:41155 (local address 127.7.90.67:55193)
0619 11:13:40.091131 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.091184 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.092592 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.093022 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.093029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.093106 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:13:40.093685 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.093701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.094421 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.094424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.095327 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.095336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.095784 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.095787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.095919 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.096431 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:13:40.096433 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:13:40.096468 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:13:40.096526 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:13:40.096767 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:13:40.096773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:13:40.096776 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.096865 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.097064 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260619 11:13:40.098958 27700 heartbeater.cc:347] Connected to a master server at 127.7.90.126:41155
I20260619 11:13:40.099176 27700 heartbeater.cc:464] Registering TS with master...
I20260619 11:13:40.099653 27700 heartbeater.cc:511] Master 127.7.90.126:41155 requested a full tablet report, sending...
I20260619 11:13:40.100616 27250 ts_manager.cc:194] Registered new tserver with Master: e844b97fc5564ec994da37169bcdc221 (127.7.90.67:40645)
I20260619 11:13:40.101147 27250 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.90.67:55193
I20260619 11:13:40.103111 7529 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:13:40.108180 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.104995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.7.90.126:41155)
0619 11:13:40.105105 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.105109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.105221 (+ 112us) server_negotiation.cc:407] Connection header received
0619 11:13:40.105296 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.105298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.105331 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.105395 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:40.105658 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.106125 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.106895 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.107088 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.107159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.107904 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:40.107907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:40.107909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:40.107918 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:40.107931 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:40.107938 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:40.107990 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.108040 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.108090 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:40.110874 27250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:13:40.113937 27250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:13:40.127197 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.121790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:40645 (local address 127.0.0.1:45978)
0619 11:13:40.122788 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.122828 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.122946 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.123393 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.123398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.123419 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:40.123561 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.123570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.124699 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.124704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.125832 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.125840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.126692 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.126697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.126798 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.126836 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.126954 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.127058 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260619 11:13:40.127262 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.122129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.7.90.67:40645)
0619 11:13:40.122559 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.122570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.123045 (+ 475us) server_negotiation.cc:407] Connection header received
0619 11:13:40.123105 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.123108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.123249 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.123402 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:40.123698 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.124598 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.125959 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.126558 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.126668 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.126847 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.126995 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.127092 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":121,"threads_started":1}
I20260619 11:13:40.127954 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.121626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.66:36499 (local address 127.0.0.1:33112)
0619 11:13:40.122777 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.122822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.122945 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.123558 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.123562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.123578 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:40.123678 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.123686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.125130 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.125133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.126268 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.126277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.127536 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.127539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.127647 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.127681 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.127756 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.127827 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260619 11:13:40.128075 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.121765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.7.90.66:36499)
0619 11:13:40.123179 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.123186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.123204 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:13:40.123257 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.123261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.123403 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.123566 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:40.123832 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.125004 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.126396 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.127169 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.127277 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.127482 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.127811 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.127914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":130,"threads_started":1}
I20260619 11:13:40.128264 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.122239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43487 (local address 127.0.0.1:37028)
0619 11:13:40.123650 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.123664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.123754 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.124165 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.124168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.124186 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:40.124261 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.124267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.125602 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.125605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.126596 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.126603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.127880 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.127883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.127960 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.127987 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.128055 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.128129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"thread_start_us":89,"threads_started":1}
I20260619 11:13:40.128630 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.122396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.7.90.65:43487)
0619 11:13:40.122848 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.122862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.123756 (+ 894us) server_negotiation.cc:407] Connection header received
0619 11:13:40.123843 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.123847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.124024 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.124197 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:40.124387 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.125461 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.126751 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.127531 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.127638 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.127822 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.128428 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.128518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":155,"threads_started":1}
I20260619 11:13:40.130666 27503 tablet_service.cc:1467] Processing CreateTablet for tablet 6acc56afe3954866897537a6e86ccf30 (DEFAULT_TABLE table=test-table [id=a5b4d107f258463eaf011e0c8ac3ef9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:40.130604 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 6acc56afe3954866897537a6e86ccf30 (DEFAULT_TABLE table=test-table [id=a5b4d107f258463eaf011e0c8ac3ef9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:40.131556 27503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acc56afe3954866897537a6e86ccf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:40.131587 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acc56afe3954866897537a6e86ccf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:40.131377 27372 tablet_service.cc:1467] Processing CreateTablet for tablet 6acc56afe3954866897537a6e86ccf30 (DEFAULT_TABLE table=test-table [id=a5b4d107f258463eaf011e0c8ac3ef9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:13:40.132210 27372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acc56afe3954866897537a6e86ccf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:40.137235 27717 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Bootstrap starting.
I20260619 11:13:40.138718 27719 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Bootstrap starting.
I20260619 11:13:40.139165 27718 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Bootstrap starting.
I20260619 11:13:40.140165 27717 tablet_bootstrap.cc:654] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:40.140789 27719 tablet_bootstrap.cc:654] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:40.140816 27718 tablet_bootstrap.cc:654] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:40.141033 27717 log.cc:824] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:40.141522 27718 log.cc:824] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:40.141577 27719 log.cc:824] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:40.142889 27717 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: No bootstrap required, opened a new log
I20260619 11:13:40.143173 27717 ts_tablet_manager.cc:1397] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:13:40.143381 27719 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: No bootstrap required, opened a new log
I20260619 11:13:40.143381 27718 tablet_bootstrap.cc:492] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: No bootstrap required, opened a new log
I20260619 11:13:40.143545 27719 ts_tablet_manager.cc:1397] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:13:40.143545 27718 ts_tablet_manager.cc:1397] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:13:40.146459 27717 raft_consensus.cc:348] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.146683 27717 raft_consensus.cc:374] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:40.146734 27717 raft_consensus.cc:729] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf423097bbda43aaa047d426efec7b5c, State: Initialized, Role: FOLLOWER
I20260619 11:13:40.147192 27717 consensus_queue.cc:260] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.147578 27718 raft_consensus.cc:348] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.147820 27718 raft_consensus.cc:374] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:40.147867 27569 heartbeater.cc:503] Master 127.7.90.126:41155 was elected leader, sending a full tablet report...
I20260619 11:13:40.147876 27718 raft_consensus.cc:729] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e844b97fc5564ec994da37169bcdc221, State: Initialized, Role: FOLLOWER
I20260619 11:13:40.148262 27717 ts_tablet_manager.cc:1428] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:13:40.148267 27718 consensus_queue.cc:260] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.147578 27719 raft_consensus.cc:348] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.148625 27719 raft_consensus.cc:374] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:40.148689 27719 raft_consensus.cc:729] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93d6a729ccd4535bd746f80679052df, State: Initialized, Role: FOLLOWER
I20260619 11:13:40.149015 27700 heartbeater.cc:503] Master 127.7.90.126:41155 was elected leader, sending a full tablet report...
I20260619 11:13:40.149176 27718 ts_tablet_manager.cc:1428] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:13:40.149127 27719 consensus_queue.cc:260] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.150096 27719 ts_tablet_manager.cc:1428] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:13:40.150539 27438 heartbeater.cc:503] Master 127.7.90.126:41155 was elected leader, sending a full tablet report...
W20260619 11:13:40.158099 27570 tablet.cc:2369] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:13:40.214864 27439 tablet.cc:2369] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:13:40.261695 27723 raft_consensus.cc:482] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:13:40.261894 27723 raft_consensus.cc:504] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.262953 27723 leader_election.cc:288] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e844b97fc5564ec994da37169bcdc221 (127.7.90.67:40645), d93d6a729ccd4535bd746f80679052df (127.7.90.65:43487)
I20260619 11:13:40.266781 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.263403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:35601 (local address 127.7.90.65:43487)
0619 11:13:40.263522 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.263528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.263541 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:13:40.263597 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.263600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.263656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.263740 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:40.264038 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.264654 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.265849 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.266222 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.266301 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.266396 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.266579 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.266650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:13:40.266805 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.263291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.90.66:51307 (local address 127.7.90.67:40645)
0619 11:13:40.263427 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.263431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.263748 (+ 317us) server_negotiation.cc:407] Connection header received
0619 11:13:40.263870 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.263874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.263924 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.264004 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:13:40.264286 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.264815 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.265847 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.266247 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.266369 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.266427 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.266622 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.266694 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:13:40.266788 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.263184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43487 (local address 127.7.90.66:35601)
0619 11:13:40.263290 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.263321 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.263414 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.263771 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.263775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.263795 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:40.263921 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.263928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.264764 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.264767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.265726 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.265735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.266348 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.266352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.266452 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.266479 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.266538 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.266622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":24}
I20260619 11:13:40.266806 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.263184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.67:40645 (local address 127.7.90.66:51307)
0619 11:13:40.263629 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.263642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:13:40.263750 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:13:40.264035 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:13:40.264038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:13:40.264055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:13:40.264179 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.264185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.264927 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.264929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.265727 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:13:40.265736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.266390 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.266394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:13:40.266483 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.266499 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:13:40.266593 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 11:13:40.266650 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":108,"thread_start_us":132,"threads_started":1}
I20260619 11:13:40.267409 27392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acc56afe3954866897537a6e86ccf30" candidate_uuid: "bf423097bbda43aaa047d426efec7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93d6a729ccd4535bd746f80679052df" is_pre_election: true
I20260619 11:13:40.267438 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acc56afe3954866897537a6e86ccf30" candidate_uuid: "bf423097bbda43aaa047d426efec7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e844b97fc5564ec994da37169bcdc221" is_pre_election: true
I20260619 11:13:40.267696 27654 raft_consensus.cc:2454] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf423097bbda43aaa047d426efec7b5c in term 0.
I20260619 11:13:40.267695 27392 raft_consensus.cc:2454] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf423097bbda43aaa047d426efec7b5c in term 0.
I20260619 11:13:40.268184 27459 leader_election.cc:302] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf423097bbda43aaa047d426efec7b5c, e844b97fc5564ec994da37169bcdc221; no voters:
I20260619 11:13:40.268450 27723 raft_consensus.cc:2781] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:13:40.268532 27723 raft_consensus.cc:482] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:13:40.268573 27723 raft_consensus.cc:3037] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:40.269944 27723 raft_consensus.cc:504] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.270234 27723 leader_election.cc:288] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [CANDIDATE]: Term 1 election: Requested vote from peers e844b97fc5564ec994da37169bcdc221 (127.7.90.67:40645), d93d6a729ccd4535bd746f80679052df (127.7.90.65:43487)
I20260619 11:13:40.270584 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acc56afe3954866897537a6e86ccf30" candidate_uuid: "bf423097bbda43aaa047d426efec7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e844b97fc5564ec994da37169bcdc221"
I20260619 11:13:40.270680 27654 raft_consensus.cc:3037] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:40.271809 27654 raft_consensus.cc:2454] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf423097bbda43aaa047d426efec7b5c in term 1.
I20260619 11:13:40.272114 27392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acc56afe3954866897537a6e86ccf30" candidate_uuid: "bf423097bbda43aaa047d426efec7b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93d6a729ccd4535bd746f80679052df"
I20260619 11:13:40.272133 27459 leader_election.cc:302] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf423097bbda43aaa047d426efec7b5c, e844b97fc5564ec994da37169bcdc221; no voters:
I20260619 11:13:40.272226 27392 raft_consensus.cc:3037] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:40.272303 27723 raft_consensus.cc:2781] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:40.272541 27723 raft_consensus.cc:686] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 1 LEADER]: Becoming Leader. State: Replica: bf423097bbda43aaa047d426efec7b5c, State: Running, Role: LEADER
I20260619 11:13:40.272846 27723 consensus_queue.cc:237] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } }
I20260619 11:13:40.273327 27392 raft_consensus.cc:2454] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf423097bbda43aaa047d426efec7b5c in term 1.
I20260619 11:13:40.275923 27249 catalog_manager.cc:5495] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf423097bbda43aaa047d426efec7b5c (127.7.90.66). New cstate: current_term: 1 leader_uuid: "bf423097bbda43aaa047d426efec7b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf423097bbda43aaa047d426efec7b5c" member_type: VOTER last_known_addr { host: "127.7.90.66" port: 36499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 } health_report { overall_health: UNKNOWN } } }
I20260619 11:13:40.313879 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.310687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.7.90.66:36499)
0619 11:13:40.310810 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:40.310814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:40.310826 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:13:40.310918 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:40.310921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:40.310974 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:40.311059 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:13:40.311273 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.311732 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.312610 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:40.312812 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:40.312895 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:40.313131 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:13:40.313570 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:13:40.313658 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:40.313709 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 11:13:40.313751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:13:40.319271 27654 raft_consensus.cc:1261] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 1 FOLLOWER]: Refusing update from remote peer bf423097bbda43aaa047d426efec7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:40.319279 27392 raft_consensus.cc:1261] T 6acc56afe3954866897537a6e86ccf30 P d93d6a729ccd4535bd746f80679052df [term 1 FOLLOWER]: Refusing update from remote peer bf423097bbda43aaa047d426efec7b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:13:40.319952 27727 consensus_queue.cc:1035] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e844b97fc5564ec994da37169bcdc221" member_type: VOTER last_known_addr { host: "127.7.90.67" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:40.320096 27723 consensus_queue.cc:1035] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93d6a729ccd4535bd746f80679052df" member_type: VOTER last_known_addr { host: "127.7.90.65" port: 43487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:13:40.337779 27250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:40.337944 27250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:13:40.340332 27250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fca24f23ccd04632a4d0085616547d1e: Sending DeleteTablet for 3 replicas of tablet 6acc56afe3954866897537a6e86ccf30
I20260619 11:13:40.341032 27503 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acc56afe3954866897537a6e86ccf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:40 UTC) from {username='slave'} at 127.0.0.1:33112
I20260619 11:13:40.341321 27372 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acc56afe3954866897537a6e86ccf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:40 UTC) from {username='slave'} at 127.0.0.1:37028
I20260619 11:13:40.341496 27740 tablet_replica.cc:330] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: stopping tablet replica
I20260619 11:13:40.341477 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acc56afe3954866897537a6e86ccf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:13:40 UTC) from {username='slave'} at 127.0.0.1:45978
I20260619 11:13:40.341809 27740 raft_consensus.cc:2229] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:13:40.341909 27742 tablet_replica.cc:330] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: stopping tablet replica
I20260619 11:13:40.342167 27742 raft_consensus.cc:2229] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:13:40.342195 27740 raft_consensus.cc:2258] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:40.342459 27742 raft_consensus.cc:2258] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:13:40.342686 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27310
I20260619 11:13:40.343626 27740 ts_tablet_manager.cc:1905] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:40.343974 27742 ts_tablet_manager.cc:1905] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:13:40.346524 27740 ts_tablet_manager.cc:1918] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:40.346524 27742 ts_tablet_manager.cc:1918] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:13:40.346616 27742 log.cc:1196] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-2/wal/wals/6acc56afe3954866897537a6e86ccf30
I20260619 11:13:40.346616 27740 log.cc:1196] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Deleting WAL directory at /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867375844525-7529-0/minicluster-data/ts-1/wal/wals/6acc56afe3954866897537a6e86ccf30
I20260619 11:13:40.346937 27742 ts_tablet_manager.cc:1939] T 6acc56afe3954866897537a6e86ccf30 P e844b97fc5564ec994da37169bcdc221: Deleting consensus metadata
I20260619 11:13:40.346952 27740 ts_tablet_manager.cc:1939] T 6acc56afe3954866897537a6e86ccf30 P bf423097bbda43aaa047d426efec7b5c: Deleting consensus metadata
I20260619 11:13:40.348068 27237 catalog_manager.cc:4841] TS e844b97fc5564ec994da37169bcdc221 (127.7.90.67:40645): tablet 6acc56afe3954866897537a6e86ccf30 (table test-table [id=a5b4d107f258463eaf011e0c8ac3ef9d]) successfully deleted
I20260619 11:13:40.348186 27235 catalog_manager.cc:4841] TS bf423097bbda43aaa047d426efec7b5c (127.7.90.66:36499): tablet 6acc56afe3954866897537a6e86ccf30 (table test-table [id=a5b4d107f258463eaf011e0c8ac3ef9d]) successfully deleted
W20260619 11:13:40.350795 27234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.90.65:43487 (error 108)
I20260619 11:13:40.351536 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27441
I20260619 11:13:40.351851 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:40.351541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.65:43487 (local address 127.0.0.1:37040)
0619 11:13:40.351688 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 11:13:40.351758 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 11:13:40.352382 27234 catalog_manager.cc:4584] TS d93d6a729ccd4535bd746f80679052df (127.7.90.65:43487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6acc56afe3954866897537a6e86ccf30: Network error: Client connection negotiation failed: client connection to 127.7.90.65:43487: connect: Connection refused (error 111)
I20260619 11:13:40.356881 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27572
I20260619 11:13:40.362258 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27219
2026-06-19T11:13:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1087 ms)
[----------] 6 tests from EncryptionPolicyTest (10170 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:40 dist-test-slave-f1t5 krb5kdc[27746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:40 dist-test-slave-f1t5 krb5kdc[27746](info): set up 2 sockets
Jun 19 11:13:40 dist-test-slave-f1t5 krb5kdc[27746](info): commencing operation
krb5kdc: starting...
W20260619 11:13:41.401736 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:41.518648 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:41.623426 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:41.623643 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:41.623699 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:41.627449 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:41.627491 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:41.627508 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:41.627522 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:41.627535 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:41.632030 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:41.632846 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:41.633776 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:41.641429 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:41.641559 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:41.641669 27759 server_base.cc:1034] running on GCE node
W20260619 11:13:41.641921 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:41.642444 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:41.643548 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:41.644730 27759 hybrid_clock.cc:648] HybridClock initialized: now 1781867621644699 us; error 40 us; skew 500 ppm
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:41.647377 27759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:41.648171 27759 webserver.cc:460] Webserver started at http://127.0.0.1:40241/ using document root <none> and password file <none>
I20260619 11:13:41.648680 27759 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:41.648761 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:41.648963 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:41.651003 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "4c5cf605d1d244ef88fbebab68116b5c"
format_stamp: "Formatted at 2026-06-19 11:13:41 on dist-test-slave-f1t5"
server_key: "0e9716f329942d5a72b6be2f6dd5ea2a"
server_key_iv: "b26e5ef43eedda582e7d9eb874327892"
server_key_version: "encryptionkey@0"
I20260619 11:13:41.651532 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "4c5cf605d1d244ef88fbebab68116b5c"
format_stamp: "Formatted at 2026-06-19 11:13:41 on dist-test-slave-f1t5"
server_key: "0e9716f329942d5a72b6be2f6dd5ea2a"
server_key_iv: "b26e5ef43eedda582e7d9eb874327892"
server_key_version: "encryptionkey@0"
I20260619 11:13:41.655170 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:41.657478 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:41.658710 27759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:41.658872 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "4c5cf605d1d244ef88fbebab68116b5c"
format_stamp: "Formatted at 2026-06-19 11:13:41 on dist-test-slave-f1t5"
server_key: "0e9716f329942d5a72b6be2f6dd5ea2a"
server_key_iv: "b26e5ef43eedda582e7d9eb874327892"
server_key_version: "encryptionkey@0"
I20260619 11:13:41.658982 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:41.690142 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:41.693696 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:41.693926 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:41.701465 27759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42783
I20260619 11:13:41.701495 27826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42783 every 8 connection(s)
I20260619 11:13:41.702430 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:41.703970 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27759
I20260619 11:13:41.704068 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:41.704371 7529 external_mini_cluster.cc:1428] Setting key 24bd3cd903be0770589c940547ffc000
I20260619 11:13:41.705215 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:41.711587 27827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c: Bootstrap starting.
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:13:41.714231 27827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:41.714920 27827 log.cc:824] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:41.716986 27827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c: No bootstrap required, opened a new log
I20260619 11:13:41.719378 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:41.705982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53659 (local address 127.0.0.1:42783)
0619 11:13:41.706391 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:41.706418 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:41.706455 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:13:41.707093 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:41.707108 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:41.707426 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:41.707709 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:41.708592 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:41.709686 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:41.710836 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:41.711288 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:41.711425 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:41.714187 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:41.714193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:41.714205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:41.714230 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:41.716183 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:41.716652 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:41.716657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:41.716662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:41.716791 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:41.717191 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:41.717194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:41.717196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:41.717700 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:41.717892 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:41.718159 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 11:13:41.718371 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260619 11:13:41.721304 27827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260619 11:13:41.721565 27827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:41.721627 27827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5cf605d1d244ef88fbebab68116b5c, State: Initialized, Role: FOLLOWER
I20260619 11:13:41.722069 27827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260619 11:13:41.722220 27827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:41.722287 27827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:41.722334 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:41.722383 27827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:41.723166 27827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260619 11:13:41.723522 27827 leader_election.cc:302] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5cf605d1d244ef88fbebab68116b5c; no voters:
I20260619 11:13:41.723831 27827 leader_election.cc:288] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:41.723940 27836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:41.724203 27836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [term 1 LEADER]: Becoming Leader. State: Replica: 4c5cf605d1d244ef88fbebab68116b5c, State: Running, Role: LEADER
I20260619 11:13:41.724737 27836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260619 11:13:41.725003 27827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:41.725939 27836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5cf605d1d244ef88fbebab68116b5c. Latest consensus state: current_term: 1 leader_uuid: "4c5cf605d1d244ef88fbebab68116b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } } }
I20260619 11:13:41.726073 27836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:41.726326 27837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5cf605d1d244ef88fbebab68116b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5cf605d1d244ef88fbebab68116b5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } } }
I20260619 11:13:41.726425 27837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:41.726531 27846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:41.729751 27846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:41.735167 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:41.724331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43247 (local address 127.0.0.1:42783)
0619 11:13:41.724445 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:41.724449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:41.725819 (+ 1370us) server_negotiation.cc:407] Connection header received
0619 11:13:41.726533 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:41.726537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:41.726589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:41.726678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:41.727572 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:41.728276 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:41.729472 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:41.729856 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:41.729952 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:41.731403 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:41.731406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:41.731409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:41.731446 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:41.733117 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:41.733745 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:41.733749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:41.733751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:41.733846 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:41.734268 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:41.734272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:41.734273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:41.734438 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:41.734648 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:41.734922 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 11:13:41.735012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:13:41.736302 27846 catalog_manager.cc:1347] Generated new cluster ID: 5e0b52ae376949e997738a1e26fe7a4d
I20260619 11:13:41.736373 27846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:41.752363 27846 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:41.753216 27846 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:41.760452 27846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c5cf605d1d244ef88fbebab68116b5c: Generated new TSK 0
I20260619 11:13:41.761092 27846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:41.767624 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27759
W20260619 11:13:41.773104 27744 connection.cc:535] client connection to 127.0.0.1:42783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42783): Connection reset by peer (error 104)
2026-06-19T11:13:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27858](info): set up 2 sockets
Jun 19 11:13:41 dist-test-slave-f1t5 krb5kdc[27858](info): commencing operation
krb5kdc: starting...
W20260619 11:13:42.789696 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:42 dist-test-slave-f1t5 krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:42.906041 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:43.011004 27871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:43.011229 27871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:43.011270 27871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:43.015242 27871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:43.015295 27871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:43.015316 27871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:43.015336 27871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:43.015352 27871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:43.020452 27871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:43.021327 27871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:43.022305 27871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:43.030651 27876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:43.030665 27871 server_base.cc:1034] running on GCE node
W20260619 11:13:43.030697 27879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:43.030663 27877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:43.031391 27871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:43.032407 27871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:43.033607 27871 hybrid_clock.cc:648] HybridClock initialized: now 1781867623033579 us; error 36 us; skew 500 ppm
Jun 19 11:13:43 dist-test-slave-f1t5 krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:43.036147 27871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:43.036929 27871 webserver.cc:460] Webserver started at http://127.0.0.1:43541/ using document root <none> and password file <none>
I20260619 11:13:43.037441 27871 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:43.037528 27871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:43.037756 27871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:43.039601 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "145bdce3843343b5b4082ef3d8cc0753"
format_stamp: "Formatted at 2026-06-19 11:13:43 on dist-test-slave-f1t5"
server_key: "ab73b971699f86a805732e082e8e880e"
server_key_iv: "b8baf4fa4db75e238cb7c14ce7f88473"
server_key_version: "encryptionkey@0"
I20260619 11:13:43.040089 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "145bdce3843343b5b4082ef3d8cc0753"
format_stamp: "Formatted at 2026-06-19 11:13:43 on dist-test-slave-f1t5"
server_key: "ab73b971699f86a805732e082e8e880e"
server_key_iv: "b8baf4fa4db75e238cb7c14ce7f88473"
server_key_version: "encryptionkey@0"
I20260619 11:13:43.043476 27871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:13:43.045666 27886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:43.046734 27871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:13:43.046876 27871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "145bdce3843343b5b4082ef3d8cc0753"
format_stamp: "Formatted at 2026-06-19 11:13:43 on dist-test-slave-f1t5"
server_key: "ab73b971699f86a805732e082e8e880e"
server_key_iv: "b8baf4fa4db75e238cb7c14ce7f88473"
server_key_version: "encryptionkey@0"
I20260619 11:13:43.046978 27871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:43.073174 27871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:43.075908 27871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:43.076095 27871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:43.082723 27871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35773
I20260619 11:13:43.082791 27938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35773 every 8 connection(s)
I20260619 11:13:43.083613 27871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:43.086288 27939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:43.091140 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27871
I20260619 11:13:43.091312 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:43.091300 27939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753: Bootstrap starting.
I20260619 11:13:43.091578 7529 external_mini_cluster.cc:1428] Setting key 8159935b43b5ac822f59042204a4a224
I20260619 11:13:43.093982 27939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:43.094815 27939 log.cc:824] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:43.096969 27939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753: No bootstrap required, opened a new log
I20260619 11:13:43.101282 27939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260619 11:13:43.101476 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:43.092880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40873 (local address 127.0.0.1:35773)
0619 11:13:43.093280 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:43.093302 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:43.093343 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:13:43.094257 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:43.094279 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:43.094693 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:43.095053 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:43.095451 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:43.096810 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:43.097802 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:43.098472 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:43.098627 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:43.099455 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:43.099460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:43.099477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:43.100072 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:43.100137 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:43.100167 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:43.100297 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:43.100447 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 11:13:43.100577 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260619 11:13:43.101573 27939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:43.101619 27939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145bdce3843343b5b4082ef3d8cc0753, State: Initialized, Role: FOLLOWER
I20260619 11:13:43.102046 27939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260619 11:13:43.102188 27939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:43.102252 27939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:43.102337 27939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:43.103168 27939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260619 11:13:43.103507 27939 leader_election.cc:302] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 145bdce3843343b5b4082ef3d8cc0753; no voters:
I20260619 11:13:43.103817 27939 leader_election.cc:288] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:43.103899 27944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:43.104116 27944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [term 1 LEADER]: Becoming Leader. State: Replica: 145bdce3843343b5b4082ef3d8cc0753, State: Running, Role: LEADER
I20260619 11:13:43.104308 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:43.104482 27944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } }
I20260619 11:13:43.105173 27939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:43.105428 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "145bdce3843343b5b4082ef3d8cc0753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } } }
I20260619 11:13:43.105559 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:43.105880 27946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 145bdce3843343b5b4082ef3d8cc0753. Latest consensus state: current_term: 1 leader_uuid: "145bdce3843343b5b4082ef3d8cc0753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145bdce3843343b5b4082ef3d8cc0753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35773 } } }
I20260619 11:13:43.105953 27946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:43.107148 27951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:43.110198 27951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:43.111217 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:43.106756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35383 (local address 127.0.0.1:35773)
0619 11:13:43.106867 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:43.106871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:43.107613 (+ 742us) server_negotiation.cc:407] Connection header received
0619 11:13:43.107788 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:43.107791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:43.107832 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:43.107879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:43.108250 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:43.108729 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:43.109567 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:43.109809 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:43.109918 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:43.110774 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:43.110777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:43.110780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:43.110800 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:43.110829 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:43.110839 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:43.110936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:43.111054 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 11:13:43.111085 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:13:43.116222 27951 catalog_manager.cc:1347] Generated new cluster ID: 4363a482e0314f22a1bf56a8134a4ba3
I20260619 11:13:43.116292 27951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:43.123541 27951 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:43.124673 27951 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:43.130189 27951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 145bdce3843343b5b4082ef3d8cc0753: Generated new TSK 0
I20260619 11:13:43.130949 27951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:43.135531 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27871
W20260619 11:13:43.141052 27856 connection.cc:535] client connection to 127.0.0.1:35773 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35773): Connection reset by peer (error 104)
2026-06-19T11:13:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:43 dist-test-slave-f1t5 krb5kdc[27970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:43 dist-test-slave-f1t5 krb5kdc[27970](info): set up 2 sockets
Jun 19 11:13:43 dist-test-slave-f1t5 krb5kdc[27970](info): commencing operation
krb5kdc: starting...
W20260619 11:13:44.177732 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:44 dist-test-slave-f1t5 krb5kdc[27970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:44.292423 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:44.399107 27983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:44.399394 27983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:44.399474 27983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:44.403439 27983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:44.403527 27983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:44.403568 27983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:44.403625 27983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:44.403661 27983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:44.408710 27983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:44.409739 27983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:44.410724 27983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:44.419281 27988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:44.419263 27989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:44.419487 27983 server_base.cc:1034] running on GCE node
W20260619 11:13:44.419564 27991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:44.420097 27983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:44.421201 27983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:44.422369 27983 hybrid_clock.cc:648] HybridClock initialized: now 1781867624422333 us; error 53 us; skew 500 ppm
Jun 19 11:13:44 dist-test-slave-f1t5 krb5kdc[27970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:44.425093 27983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:44.425982 27983 webserver.cc:460] Webserver started at http://127.0.0.1:35057/ using document root <none> and password file <none>
I20260619 11:13:44.426482 27983 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:44.426543 27983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:44.426705 27983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:44.428689 27983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "1cc24ea812054c4b93c025f9d9bac144"
format_stamp: "Formatted at 2026-06-19 11:13:44 on dist-test-slave-f1t5"
server_key: "40e001f2fde7db282738dbd062bcc950"
server_key_iv: "43da7d3e2d9537df2453086c565c105f"
server_key_version: "encryptionkey@0"
I20260619 11:13:44.429203 27983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "1cc24ea812054c4b93c025f9d9bac144"
format_stamp: "Formatted at 2026-06-19 11:13:44 on dist-test-slave-f1t5"
server_key: "40e001f2fde7db282738dbd062bcc950"
server_key_iv: "43da7d3e2d9537df2453086c565c105f"
server_key_version: "encryptionkey@0"
I20260619 11:13:44.432817 27983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:13:44.435178 27998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:44.436336 27983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:13:44.436478 27983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "1cc24ea812054c4b93c025f9d9bac144"
format_stamp: "Formatted at 2026-06-19 11:13:44 on dist-test-slave-f1t5"
server_key: "40e001f2fde7db282738dbd062bcc950"
server_key_iv: "43da7d3e2d9537df2453086c565c105f"
server_key_version: "encryptionkey@0"
I20260619 11:13:44.436575 27983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:44.441061 27983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:44.443760 27983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:44.443961 27983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:44.450451 27983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43853
I20260619 11:13:44.450481 28050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43853 every 8 connection(s)
I20260619 11:13:44.451274 27983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:44.453871 28051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:44.457625 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 27983
I20260619 11:13:44.457767 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:44.458051 7529 external_mini_cluster.cc:1428] Setting key 6aca2bd8d7cdf1020d12f1fa4896e37a
I20260619 11:13:44.459372 28051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144: Bootstrap starting.
I20260619 11:13:44.462057 28051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:44.462961 28051 log.cc:824] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:44.463593 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:44.459412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46615 (local address 127.0.0.1:43853)
0619 11:13:44.459857 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:44.459874 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:44.459914 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:13:44.460769 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:44.460793 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:44.461132 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:44.461485 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:44.462278 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:44.462284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:44.462301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:44.462316 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:44.462385 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:44.462420 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:44.462525 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:44.462633 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 11:13:44.462779 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260619 11:13:44.465045 28051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144: No bootstrap required, opened a new log
I20260619 11:13:44.466437 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:44.468981 28051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260619 11:13:44.469233 28051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:44.469308 28051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc24ea812054c4b93c025f9d9bac144, State: Initialized, Role: FOLLOWER
I20260619 11:13:44.469820 28051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260619 11:13:44.469995 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:44.468396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57013 (local address 127.0.0.1:43853)
0619 11:13:44.468522 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:44.468527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:44.468739 (+ 212us) server_negotiation.cc:407] Connection header received
0619 11:13:44.468807 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:44.468810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:44.468842 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:44.468924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:44.469618 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:44.469623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:44.469626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:44.469639 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:44.469670 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:44.469681 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:44.469751 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:44.469851 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 11:13:44.469879 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:13:44.470072 28051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:44.470147 28051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:44.470242 28051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:44.471251 28051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260619 11:13:44.471586 28051 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc24ea812054c4b93c025f9d9bac144; no voters:
I20260619 11:13:44.471875 28051 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:44.471972 28061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:44.472247 28061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc24ea812054c4b93c025f9d9bac144, State: Running, Role: LEADER
I20260619 11:13:44.472582 28061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } }
I20260619 11:13:44.473133 28051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:44.474357 28063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc24ea812054c4b93c025f9d9bac144. Latest consensus state: current_term: 1 leader_uuid: "1cc24ea812054c4b93c025f9d9bac144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } } }
I20260619 11:13:44.474349 28062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc24ea812054c4b93c025f9d9bac144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc24ea812054c4b93c025f9d9bac144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43853 } } }
I20260619 11:13:44.474488 28063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:44.474489 28062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:44.474884 28070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:44.478096 28070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:44.483451 28070 catalog_manager.cc:1347] Generated new cluster ID: f76acd90ff49406290d119647cca32ab
I20260619 11:13:44.483522 28070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:44.495571 28070 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:44.495885 28070 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:44.506569 28070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc24ea812054c4b93c025f9d9bac144: Generated new TSK 0
I20260619 11:13:44.507205 28070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:44.513290 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 27983
2026-06-19T11:13:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:44 dist-test-slave-f1t5 krb5kdc[28082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:44 dist-test-slave-f1t5 krb5kdc[28082](info): set up 2 sockets
Jun 19 11:13:44 dist-test-slave-f1t5 krb5kdc[28082](info): commencing operation
krb5kdc: starting...
W20260619 11:13:45.545823 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:45.656188 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:45.764266 28095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:45.764511 28095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:45.764551 28095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:45.768548 28095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:45.768599 28095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:45.768620 28095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:45.768635 28095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:45.768675 28095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:45.773799 28095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:45.774827 28095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:45.775835 28095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:45.784061 28100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:45.784081 28103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:45.784057 28101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:45.784307 28095 server_base.cc:1034] running on GCE node
I20260619 11:13:45.785094 28095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:45.786360 28095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:45.787528 28095 hybrid_clock.cc:648] HybridClock initialized: now 1781867625787489 us; error 55 us; skew 500 ppm
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:45.793711 28095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:45.794767 28095 webserver.cc:460] Webserver started at http://127.0.0.1:43123/ using document root <none> and password file <none>
I20260619 11:13:45.795387 28095 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:45.795481 28095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:45.795723 28095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:45.798030 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "86601ca2f820471a97d33e60495efdd8"
format_stamp: "Formatted at 2026-06-19 11:13:45 on dist-test-slave-f1t5"
server_key: "bf8c6a3a016bf4fcf175878230913204"
server_key_iv: "754b57b133b92d2a3e5d072f4b8bd84d"
server_key_version: "encryptionkey@0"
I20260619 11:13:45.798640 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "86601ca2f820471a97d33e60495efdd8"
format_stamp: "Formatted at 2026-06-19 11:13:45 on dist-test-slave-f1t5"
server_key: "bf8c6a3a016bf4fcf175878230913204"
server_key_iv: "754b57b133b92d2a3e5d072f4b8bd84d"
server_key_version: "encryptionkey@0"
I20260619 11:13:45.802665 28095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:13:45.805687 28110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:45.806962 28095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:45.807111 28095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "86601ca2f820471a97d33e60495efdd8"
format_stamp: "Formatted at 2026-06-19 11:13:45 on dist-test-slave-f1t5"
server_key: "bf8c6a3a016bf4fcf175878230913204"
server_key_iv: "754b57b133b92d2a3e5d072f4b8bd84d"
server_key_version: "encryptionkey@0"
I20260619 11:13:45.807248 28095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:45.825435 28095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:45.828403 28095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:45.828619 28095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:45.835683 28095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33903
I20260619 11:13:45.835704 28162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33903 every 8 connection(s)
I20260619 11:13:45.836665 28095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:45.839378 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:45.842214 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28095
I20260619 11:13:45.842347 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:45.842643 7529 external_mini_cluster.cc:1428] Setting key 95a640102b41ded6db5fada81abb182e
I20260619 11:13:45.845274 28163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: Bootstrap starting.
I20260619 11:13:45.848111 28163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:45.849123 28163 log.cc:824] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:45.851572 28163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: No bootstrap required, opened a new log
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:13:45.856813 28163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } }
I20260619 11:13:45.857043 28163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:45.857079 28163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86601ca2f820471a97d33e60495efdd8, State: Initialized, Role: FOLLOWER
I20260619 11:13:45.857597 28163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } }
I20260619 11:13:45.857735 28163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:45.857831 28163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:45.857928 28163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:45.858842 28163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } }
I20260619 11:13:45.859198 28163 leader_election.cc:302] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86601ca2f820471a97d33e60495efdd8; no voters:
I20260619 11:13:45.859560 28163 leader_election.cc:288] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:45.859679 28168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:45.859858 28168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [term 1 LEADER]: Becoming Leader. State: Replica: 86601ca2f820471a97d33e60495efdd8, State: Running, Role: LEADER
I20260619 11:13:45.860153 28168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } }
I20260619 11:13:45.860734 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:45.844033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41747 (local address 127.0.0.1:33903)
0619 11:13:45.844478 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:45.844492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:45.844526 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:13:45.845146 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:45.845162 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:45.845606 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:45.845939 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:45.846843 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:45.848663 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:45.850091 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:45.850769 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:45.850920 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:45.854616 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:45.854625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:45.854645 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:45.854680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:45.857075 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:45.857635 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:45.857641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:45.857647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:45.857850 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:45.858198 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:45.858201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:45.858203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:45.858616 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:45.858834 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:45.859459 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 11:13:45.859804 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260619 11:13:45.860905 28163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:45.861757 28169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86601ca2f820471a97d33e60495efdd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } } }
I20260619 11:13:45.861901 28169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:45.862123 28169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86601ca2f820471a97d33e60495efdd8. Latest consensus state: current_term: 1 leader_uuid: "86601ca2f820471a97d33e60495efdd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86601ca2f820471a97d33e60495efdd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33903 } } }
I20260619 11:13:45.862201 28169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:45.863829 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 11:13:45.866216 28184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:13:45.866282 28184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:13:45.866413 28176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:45.867097 28176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:45.872223 28176 catalog_manager.cc:1347] Generated new cluster ID: 1834e14d162144209a3ed24e3863821d
I20260619 11:13:45.872289 28176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:45.882272 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:45.870030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35913 (local address 127.0.0.1:33903)
0619 11:13:45.870140 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:45.870145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:45.873862 (+ 3717us) server_negotiation.cc:407] Connection header received
0619 11:13:45.873982 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:45.873985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:45.874037 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:45.874136 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:45.874898 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:45.875739 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:45.876847 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:45.877235 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:45.877348 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:45.878898 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:45.878902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:45.878905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:45.878933 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:45.880599 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:45.881047 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:45.881050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:45.881052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:45.881134 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:45.881404 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:45.881407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:45.881409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:45.881633 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:45.881770 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:45.881987 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 11:13:45.882101 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:13:45.909971 28176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:45.911496 28176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:45.917878 28176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86601ca2f820471a97d33e60495efdd8: Generated new TSK 0
I20260619 11:13:45.918880 28176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:45.926393 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28095
2026-06-19T11:13:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28194](info): set up 2 sockets
Jun 19 11:13:45 dist-test-slave-f1t5 krb5kdc[28194](info): commencing operation
krb5kdc: starting...
W20260619 11:13:46.961807 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:47 dist-test-slave-f1t5 krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:47.080461 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:47.186177 28207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:47.186436 28207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:47.186508 28207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:47.190368 28207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:47.190436 28207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:47.190472 28207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:47.190507 28207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:47.190542 28207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:47.195444 28207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:47.196375 28207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:47.197345 28207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:47.205287 28212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:47.205366 28207 server_base.cc:1034] running on GCE node
W20260619 11:13:47.205288 28213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:47.205650 28215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:47.206202 28207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:47.207126 28207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:47.208278 28207 hybrid_clock.cc:648] HybridClock initialized: now 1781867627208263 us; error 42 us; skew 500 ppm
Jun 19 11:13:47 dist-test-slave-f1t5 krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:47.210984 28207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:47.211795 28207 webserver.cc:460] Webserver started at http://127.0.0.1:35205/ using document root <none> and password file <none>
I20260619 11:13:47.212294 28207 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:47.212347 28207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:47.212548 28207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:47.214478 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "9073c61ba10940a3967a5deed7ca836c"
format_stamp: "Formatted at 2026-06-19 11:13:47 on dist-test-slave-f1t5"
server_key: "71d025d717099d9a7aee7be94f12fc5a"
server_key_iv: "ddd189e202f5554be5d2f0f5855617e2"
server_key_version: "encryptionkey@0"
I20260619 11:13:47.215003 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "9073c61ba10940a3967a5deed7ca836c"
format_stamp: "Formatted at 2026-06-19 11:13:47 on dist-test-slave-f1t5"
server_key: "71d025d717099d9a7aee7be94f12fc5a"
server_key_iv: "ddd189e202f5554be5d2f0f5855617e2"
server_key_version: "encryptionkey@0"
I20260619 11:13:47.218420 28207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:13:47.220634 28222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:47.221783 28207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:47.221905 28207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "9073c61ba10940a3967a5deed7ca836c"
format_stamp: "Formatted at 2026-06-19 11:13:47 on dist-test-slave-f1t5"
server_key: "71d025d717099d9a7aee7be94f12fc5a"
server_key_iv: "ddd189e202f5554be5d2f0f5855617e2"
server_key_version: "encryptionkey@0"
I20260619 11:13:47.222021 28207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:47.233868 28207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:47.236927 28207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:47.237114 28207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:47.243857 28207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46479
I20260619 11:13:47.243881 28274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46479 every 8 connection(s)
I20260619 11:13:47.244716 28207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:47.245847 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28207
I20260619 11:13:47.245980 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:47.246246 7529 external_mini_cluster.cc:1428] Setting key 5bfa0ffd3d23b7b050c451c36538d670
I20260619 11:13:47.247519 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:47.252564 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c: Bootstrap starting.
I20260619 11:13:47.254355 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:47.247521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48191 (local address 127.0.0.1:46479)
0619 11:13:47.247921 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:47.247934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:47.247967 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:13:47.248541 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:47.248555 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:47.248882 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:47.249215 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:47.249636 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:47.250534 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:47.251377 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:47.251812 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:47.252013 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:47.252822 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:47.252827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:47.252838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:47.253187 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:47.253235 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:47.253255 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:47.253363 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:47.253486 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 11:13:47.253719 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260619 11:13:47.254741 28275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:47.255514 28275 log.cc:824] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:47.257011 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:47.257351 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c: No bootstrap required, opened a new log
I20260619 11:13:47.260993 28275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260619 11:13:47.261245 28275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:47.261303 28275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9073c61ba10940a3967a5deed7ca836c, State: Initialized, Role: FOLLOWER
I20260619 11:13:47.261771 28275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260619 11:13:47.261960 28275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:47.262044 28275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:47.262181 28275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:47.263027 28275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260619 11:13:47.263341 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:47.259130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48633 (local address 127.0.0.1:46479)
0619 11:13:47.259247 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:47.259251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:47.259426 (+ 175us) server_negotiation.cc:407] Connection header received
0619 11:13:47.259531 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:47.259533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:47.259567 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:47.259628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:47.259907 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:47.260685 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:47.261760 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:47.262059 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:47.262199 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:47.262944 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:47.262948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:47.262951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:47.262962 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:47.262988 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:47.263003 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:47.263077 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:47.263169 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:13:47.263214 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:13:47.263410 28275 leader_election.cc:302] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9073c61ba10940a3967a5deed7ca836c; no voters:
I20260619 11:13:47.263731 28275 leader_election.cc:288] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:47.263818 28285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:47.264010 28285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [term 1 LEADER]: Becoming Leader. State: Replica: 9073c61ba10940a3967a5deed7ca836c, State: Running, Role: LEADER
I20260619 11:13:47.264299 28285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260619 11:13:47.264896 28275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:47.265383 28285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9073c61ba10940a3967a5deed7ca836c. Latest consensus state: current_term: 1 leader_uuid: "9073c61ba10940a3967a5deed7ca836c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260619 11:13:47.265537 28285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:47.265818 28286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9073c61ba10940a3967a5deed7ca836c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9073c61ba10940a3967a5deed7ca836c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260619 11:13:47.265916 28286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:47.265937 28291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:47.268891 28291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:47.274106 28291 catalog_manager.cc:1347] Generated new cluster ID: e296fa7546374405a7b16fdee2250305
I20260619 11:13:47.274165 28291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:47.287606 28291 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:47.288479 28291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:47.293727 28291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9073c61ba10940a3967a5deed7ca836c: Generated new TSK 0
I20260619 11:13:47.294322 28291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:47.298913 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28207
2026-06-19T11:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:47 dist-test-slave-f1t5 krb5kdc[28306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:47 dist-test-slave-f1t5 krb5kdc[28306](info): set up 2 sockets
Jun 19 11:13:47 dist-test-slave-f1t5 krb5kdc[28306](info): commencing operation
krb5kdc: starting...
W20260619 11:13:48.325676 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:48 dist-test-slave-f1t5 krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:48.442641 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:48.548017 28319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:48.548316 28319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:48.548388 28319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:48.552425 28319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:48.552512 28319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:48.552562 28319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:48.552616 28319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:48.552651 28319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:48.557864 28319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:48.558918 28319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:48.559932 28319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:48.568279 28327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:48.568286 28325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:48.568418 28319 server_base.cc:1034] running on GCE node
W20260619 11:13:48.568372 28324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:48.569123 28319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:48.570226 28319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:48.571411 28319 hybrid_clock.cc:648] HybridClock initialized: now 1781867628571385 us; error 44 us; skew 500 ppm
Jun 19 11:13:48 dist-test-slave-f1t5 krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:48.574033 28319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:48.574946 28319 webserver.cc:460] Webserver started at http://127.0.0.1:43823/ using document root <none> and password file <none>
I20260619 11:13:48.575489 28319 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:48.575577 28319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:48.575809 28319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:48.577905 28319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "480f99d2f32241de8943cf698227ccf5"
format_stamp: "Formatted at 2026-06-19 11:13:48 on dist-test-slave-f1t5"
server_key: "87fd8a3104c7e18fb2b9d1d75b3ecc30"
server_key_iv: "ffe3b8c0cc72dfe283314d01646d8b05"
server_key_version: "encryptionkey@0"
I20260619 11:13:48.578446 28319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "480f99d2f32241de8943cf698227ccf5"
format_stamp: "Formatted at 2026-06-19 11:13:48 on dist-test-slave-f1t5"
server_key: "87fd8a3104c7e18fb2b9d1d75b3ecc30"
server_key_iv: "ffe3b8c0cc72dfe283314d01646d8b05"
server_key_version: "encryptionkey@0"
I20260619 11:13:48.581985 28319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:13:48.584201 28334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:48.585306 28319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:48.585444 28319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "480f99d2f32241de8943cf698227ccf5"
format_stamp: "Formatted at 2026-06-19 11:13:48 on dist-test-slave-f1t5"
server_key: "87fd8a3104c7e18fb2b9d1d75b3ecc30"
server_key_iv: "ffe3b8c0cc72dfe283314d01646d8b05"
server_key_version: "encryptionkey@0"
I20260619 11:13:48.585577 28319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:48.589975 28319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:48.592617 28319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:48.592814 28319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:48.599474 28319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32859
I20260619 11:13:48.599524 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32859 every 8 connection(s)
I20260619 11:13:48.600381 28319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:48.603024 28387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:48.607863 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28319
I20260619 11:13:48.608021 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:48.608268 28387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5: Bootstrap starting.
I20260619 11:13:48.608300 7529 external_mini_cluster.cc:1428] Setting key add7a01b2eedcba59893fbfd7114e61a
I20260619 11:13:48.610972 28387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:48.611815 28387 log.cc:824] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:48.613726 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:48.609579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37897 (local address 127.0.0.1:32859)
0619 11:13:48.610021 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:48.610039 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:48.610079 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:13:48.610859 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:48.610883 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:48.611279 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:48.611613 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:48.612382 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:48.612386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:48.612399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:48.612407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:48.612453 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:48.612471 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:48.612596 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:48.612696 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 11:13:48.612837 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260619 11:13:48.613957 28387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5: No bootstrap required, opened a new log
I20260619 11:13:48.616387 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:48.617828 28387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 11:13:48.618046 28387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:48.618111 28387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480f99d2f32241de8943cf698227ccf5, State: Initialized, Role: FOLLOWER
I20260619 11:13:48.618645 28387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 11:13:48.618805 28387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:48.618906 28387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:48.619036 28387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:48.619834 28387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 11:13:48.620090 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:48.618430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59695 (local address 127.0.0.1:32859)
0619 11:13:48.618552 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:48.618556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:48.618748 (+ 192us) server_negotiation.cc:407] Connection header received
0619 11:13:48.618872 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:48.618875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:48.618905 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:48.618999 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:48.619733 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:48.619736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:48.619738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:48.619749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:48.619774 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:48.619785 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:48.619875 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:48.619945 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 11:13:48.619981 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:13:48.620203 28387 leader_election.cc:302] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 480f99d2f32241de8943cf698227ccf5; no voters:
I20260619 11:13:48.620538 28387 leader_election.cc:288] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:48.620604 28397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:48.620802 28397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [term 1 LEADER]: Becoming Leader. State: Replica: 480f99d2f32241de8943cf698227ccf5, State: Running, Role: LEADER
I20260619 11:13:48.621111 28397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } }
I20260619 11:13:48.621712 28387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:48.622242 28399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480f99d2f32241de8943cf698227ccf5. Latest consensus state: current_term: 1 leader_uuid: "480f99d2f32241de8943cf698227ccf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } } }
I20260619 11:13:48.622438 28399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:48.622654 28398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "480f99d2f32241de8943cf698227ccf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480f99d2f32241de8943cf698227ccf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32859 } } }
I20260619 11:13:48.622737 28398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:48.622884 28406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:48.625933 28406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:48.630558 28406 catalog_manager.cc:1347] Generated new cluster ID: 89d0dc07db074ce491442b8aa2e8ec6c
I20260619 11:13:48.630614 28406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:48.652701 28406 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:48.653021 28406 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:48.664036 28406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 480f99d2f32241de8943cf698227ccf5: Generated new TSK 0
I20260619 11:13:48.664805 28406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:48.670164 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28319
2026-06-19T11:13:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:48 dist-test-slave-f1t5 krb5kdc[28418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:48 dist-test-slave-f1t5 krb5kdc[28418](info): set up 2 sockets
Jun 19 11:13:48 dist-test-slave-f1t5 krb5kdc[28418](info): commencing operation
krb5kdc: starting...
W20260619 11:13:49.693815 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:49 dist-test-slave-f1t5 krb5kdc[28418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:49.808710 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:49.922452 28431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:49.922734 28431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:49.922847 28431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:49.926820 28431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:49.926900 28431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:49.926941 28431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:49.927000 28431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:49.927044 28431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:49.932217 28431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:49.933231 28431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:49.934298 28431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:49.942377 28437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:49.942394 28436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:49.942394 28439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:49.942718 28431 server_base.cc:1034] running on GCE node
I20260619 11:13:49.943267 28431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:49.944315 28431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:49.945505 28431 hybrid_clock.cc:648] HybridClock initialized: now 1781867629945494 us; error 27 us; skew 500 ppm
Jun 19 11:13:49 dist-test-slave-f1t5 krb5kdc[28418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:49.948104 28431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:49.948918 28431 webserver.cc:460] Webserver started at http://127.0.0.1:35403/ using document root <none> and password file <none>
I20260619 11:13:49.949460 28431 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:49.949572 28431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:49.949793 28431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:49.951676 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "343bec2b4bf043a588630db502c04551"
format_stamp: "Formatted at 2026-06-19 11:13:49 on dist-test-slave-f1t5"
server_key: "937c659da85189893ddc8e2441c43257"
server_key_iv: "24f3d41647983c31accd604367916d99"
server_key_version: "encryptionkey@0"
I20260619 11:13:49.952231 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "343bec2b4bf043a588630db502c04551"
format_stamp: "Formatted at 2026-06-19 11:13:49 on dist-test-slave-f1t5"
server_key: "937c659da85189893ddc8e2441c43257"
server_key_iv: "24f3d41647983c31accd604367916d99"
server_key_version: "encryptionkey@0"
I20260619 11:13:49.955667 28431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:49.957918 28446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:49.958993 28431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:49.959131 28431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "343bec2b4bf043a588630db502c04551"
format_stamp: "Formatted at 2026-06-19 11:13:49 on dist-test-slave-f1t5"
server_key: "937c659da85189893ddc8e2441c43257"
server_key_iv: "24f3d41647983c31accd604367916d99"
server_key_version: "encryptionkey@0"
I20260619 11:13:49.959237 28431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:49.969659 28431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:49.972050 28431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:49.972227 28431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:49.979023 28431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34649
I20260619 11:13:49.979035 28498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34649 every 8 connection(s)
I20260619 11:13:49.979836 28431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:49.982477 28499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:49.987672 28499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551: Bootstrap starting.
I20260619 11:13:49.988001 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28431
I20260619 11:13:49.988129 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:49.988456 7529 external_mini_cluster.cc:1428] Setting key b9564fb7827ba3a317f6a40e6bee187d
I20260619 11:13:49.990406 28499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:49.991125 28499 log.cc:824] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:49.993088 28499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551: No bootstrap required, opened a new log
Jun 19 11:13:49 dist-test-slave-f1t5 krb5kdc[28418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:13:49.997687 28499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } }
I20260619 11:13:49.997922 28499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:49.998014 28499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343bec2b4bf043a588630db502c04551, State: Initialized, Role: FOLLOWER
I20260619 11:13:49.998445 28499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } }
I20260619 11:13:49.998593 28499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:49.998668 28499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:49.998773 28499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:49.999619 28499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } }
I20260619 11:13:49.999950 28499 leader_election.cc:302] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 343bec2b4bf043a588630db502c04551; no voters:
I20260619 11:13:50.000303 28499 leader_election.cc:288] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:50.000411 28504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:50.000607 28504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [term 1 LEADER]: Becoming Leader. State: Replica: 343bec2b4bf043a588630db502c04551, State: Running, Role: LEADER
I20260619 11:13:50.000931 28504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } }
I20260619 11:13:50.001600 28499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:50.002508 28505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "343bec2b4bf043a588630db502c04551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } } }
I20260619 11:13:50.002645 28505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:50.002978 28506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343bec2b4bf043a588630db502c04551. Latest consensus state: current_term: 1 leader_uuid: "343bec2b4bf043a588630db502c04551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343bec2b4bf043a588630db502c04551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34649 } } }
I20260619 11:13:50.003139 28506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:50.003455 28513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:50.003693 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:49.989767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:33347 (local address 127.0.0.1:34649)
0619 11:13:49.990133 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:49.990148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:49.990180 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:13:49.990893 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:49.990910 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:49.991213 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:49.991548 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:49.992377 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:49.993382 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:49.994578 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:49.995008 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:49.995123 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:49.998093 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:49.998098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:49.998133 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:49.998162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:50.000016 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:50.000692 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:50.000698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:50.000703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:50.000848 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:50.001303 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:50.001306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:50.001308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:50.001802 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:50.002026 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:50.002368 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 11:13:50.002615 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260619 11:13:50.006531 28513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:50.006901 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:50.012686 28513 catalog_manager.cc:1347] Generated new cluster ID: ee85b14adf744fafae537c022ea846e8
I20260619 11:13:50.012745 28513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:50.019013 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:50.009178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:57353 (local address 127.0.0.1:34649)
0619 11:13:50.009276 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:50.009280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:50.009508 (+ 228us) server_negotiation.cc:407] Connection header received
0619 11:13:50.009640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:50.009643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:50.009683 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:50.009782 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:50.010383 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:50.011029 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:50.012067 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:50.012458 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:50.012586 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:50.015200 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:50.015204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:50.015208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:50.015235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:50.017023 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:50.017553 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:50.017557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:50.017559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:50.017667 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:50.018014 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:50.018018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:50.018019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:50.018256 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:50.018424 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:50.018687 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 11:13:50.018794 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:13:50.022995 28513 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:50.024200 28513 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:50.031451 28513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 343bec2b4bf043a588630db502c04551: Generated new TSK 0
I20260619 11:13:50.032209 28513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:50.037982 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28431
W20260619 11:13:50.043259 28416 connection.cc:535] client connection to 127.0.0.1:34649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34649): Connection reset by peer (error 104)
2026-06-19T11:13:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:50 dist-test-slave-f1t5 krb5kdc[28530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:50 dist-test-slave-f1t5 krb5kdc[28530](info): set up 2 sockets
Jun 19 11:13:50 dist-test-slave-f1t5 krb5kdc[28530](info): commencing operation
krb5kdc: starting...
W20260619 11:13:51.074043 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:51 dist-test-slave-f1t5 krb5kdc[28530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:51.188292 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:51.290266 28543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:51.290496 28543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:51.290534 28543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:51.294423 28543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:51.294473 28543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:51.294492 28543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:51.294508 28543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:51.294523 28543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:51.299537 28543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:51.300468 28543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:51.301416 28543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:51.309409 28549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:51.309501 28543 server_base.cc:1034] running on GCE node
W20260619 11:13:51.309425 28551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:51.309459 28548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:51.310206 28543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:51.311193 28543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:51.312348 28543 hybrid_clock.cc:648] HybridClock initialized: now 1781867631312331 us; error 29 us; skew 500 ppm
Jun 19 11:13:51 dist-test-slave-f1t5 krb5kdc[28530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:51.317811 28543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:51.318586 28543 webserver.cc:460] Webserver started at http://127.0.0.1:37765/ using document root <none> and password file <none>
I20260619 11:13:51.319108 28543 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:51.319162 28543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:51.319375 28543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:51.321219 28543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "22271c3a920342af978abc2862304dde"
format_stamp: "Formatted at 2026-06-19 11:13:51 on dist-test-slave-f1t5"
server_key: "30e3107f481e5363aa0c75c7d42345a7"
server_key_iv: "49ef281f67fc8fd76693e28e920144fd"
server_key_version: "encryptionkey@0"
I20260619 11:13:51.321777 28543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "22271c3a920342af978abc2862304dde"
format_stamp: "Formatted at 2026-06-19 11:13:51 on dist-test-slave-f1t5"
server_key: "30e3107f481e5363aa0c75c7d42345a7"
server_key_iv: "49ef281f67fc8fd76693e28e920144fd"
server_key_version: "encryptionkey@0"
I20260619 11:13:51.325222 28543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 11:13:51.327587 28558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:51.328703 28543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:51.328833 28543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "22271c3a920342af978abc2862304dde"
format_stamp: "Formatted at 2026-06-19 11:13:51 on dist-test-slave-f1t5"
server_key: "30e3107f481e5363aa0c75c7d42345a7"
server_key_iv: "49ef281f67fc8fd76693e28e920144fd"
server_key_version: "encryptionkey@0"
I20260619 11:13:51.328934 28543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:51.338683 28543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:51.341148 28543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:51.341334 28543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:51.347862 28543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45981
I20260619 11:13:51.347891 28610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45981 every 8 connection(s)
I20260619 11:13:51.348750 28543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:51.351162 28611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:51.352969 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28543
I20260619 11:13:51.353142 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:51.353457 7529 external_mini_cluster.cc:1428] Setting key 1ac93a556234794980265fedfe096f8d
I20260619 11:13:51.357044 28611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde: Bootstrap starting.
I20260619 11:13:51.359463 28611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:51.360250 28611 log.cc:824] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:51.362342 28611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde: No bootstrap required, opened a new log
I20260619 11:13:51.362964 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:51.354844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:59745 (local address 127.0.0.1:45981)
0619 11:13:51.355242 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:51.355260 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:51.355299 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:13:51.356078 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:51.356103 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:51.356524 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:51.356855 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:51.357291 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:51.358740 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:51.359612 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:51.360265 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:51.360420 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:51.361175 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:51.361180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:51.361195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:51.361656 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:51.361715 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:51.361742 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:51.361866 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:51.362009 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 11:13:51.362139 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260619 11:13:51.365694 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:51.366273 28611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } }
I20260619 11:13:51.366464 28611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:51.366526 28611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22271c3a920342af978abc2862304dde, State: Initialized, Role: FOLLOWER
I20260619 11:13:51.366950 28611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } }
I20260619 11:13:51.367094 28611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:51.367175 28611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:51.367264 28611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:51.368332 28611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } }
I20260619 11:13:51.368705 28611 leader_election.cc:302] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22271c3a920342af978abc2862304dde; no voters:
I20260619 11:13:51.368992 28611 leader_election.cc:288] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:51.369144 28621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:51.369375 28621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [term 1 LEADER]: Becoming Leader. State: Replica: 22271c3a920342af978abc2862304dde, State: Running, Role: LEADER
I20260619 11:13:51.369669 28621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } }
I20260619 11:13:51.370216 28611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:51.370743 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22271c3a920342af978abc2862304dde. Latest consensus state: current_term: 1 leader_uuid: "22271c3a920342af978abc2862304dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } } }
I20260619 11:13:51.370883 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:51.371277 28622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22271c3a920342af978abc2862304dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22271c3a920342af978abc2862304dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45981 } } }
I20260619 11:13:51.371426 28622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:51.372658 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:51.367681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:56429 (local address 127.0.0.1:45981)
0619 11:13:51.367778 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:51.367782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:51.368001 (+ 219us) server_negotiation.cc:407] Connection header received
0619 11:13:51.368064 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:51.368067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:51.368108 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:51.368191 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:51.368407 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:51.369186 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:51.370352 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:51.370729 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:51.370823 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:51.371928 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:51.371932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:51.371935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:51.371949 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:51.371977 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:51.371990 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:51.372074 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:51.372301 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:13:51.372332 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":38}
I20260619 11:13:51.371574 28627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:51.375221 28627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:51.379786 28627 catalog_manager.cc:1347] Generated new cluster ID: a45cacaae8c84102954397c738159f09
I20260619 11:13:51.379846 28627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:51.385165 28627 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:51.386013 28627 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:51.398173 28627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22271c3a920342af978abc2862304dde: Generated new TSK 0
I20260619 11:13:51.398795 28627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:51.402205 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28543
W20260619 11:13:51.407392 28528 connection.cc:535] client connection to 127.0.0.1:45981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45981): Connection reset by peer (error 104)
2026-06-19T11:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:51 dist-test-slave-f1t5 krb5kdc[28642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:51 dist-test-slave-f1t5 krb5kdc[28642](info): set up 2 sockets
Jun 19 11:13:51 dist-test-slave-f1t5 krb5kdc[28642](info): commencing operation
krb5kdc: starting...
W20260619 11:13:52.429812 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:52 dist-test-slave-f1t5 krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:52.546159 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:52.651595 28655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:52.651854 28655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:52.651896 28655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:52.655905 28655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:52.655956 28655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:52.655975 28655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:52.655992 28655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:52.656041 28655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:52.661126 28655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:52.662169 28655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:52.663151 28655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:52.671324 28663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:52.671334 28660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:52.671334 28661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:52.671654 28655 server_base.cc:1034] running on GCE node
I20260619 11:13:52.672224 28655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:52.673264 28655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:52.674448 28655 hybrid_clock.cc:648] HybridClock initialized: now 1781867632674426 us; error 31 us; skew 500 ppm
Jun 19 11:13:52 dist-test-slave-f1t5 krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:52.677022 28655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:52.677896 28655 webserver.cc:460] Webserver started at http://127.0.0.1:40281/ using document root <none> and password file <none>
I20260619 11:13:52.678439 28655 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:52.678529 28655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:52.678746 28655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:52.680727 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "e0bdeab7d5c54d92a931db08ddd7c13f"
format_stamp: "Formatted at 2026-06-19 11:13:52 on dist-test-slave-f1t5"
server_key: "6ecccd67878c77caea4c85d3ff4bfebb"
server_key_iv: "3c1888c1837faee5b405f49687f42098"
server_key_version: "encryptionkey@0"
I20260619 11:13:52.681334 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "e0bdeab7d5c54d92a931db08ddd7c13f"
format_stamp: "Formatted at 2026-06-19 11:13:52 on dist-test-slave-f1t5"
server_key: "6ecccd67878c77caea4c85d3ff4bfebb"
server_key_iv: "3c1888c1837faee5b405f49687f42098"
server_key_version: "encryptionkey@0"
I20260619 11:13:52.685002 28655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:13:52.687525 28670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:52.688705 28655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:52.688861 28655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "e0bdeab7d5c54d92a931db08ddd7c13f"
format_stamp: "Formatted at 2026-06-19 11:13:52 on dist-test-slave-f1t5"
server_key: "6ecccd67878c77caea4c85d3ff4bfebb"
server_key_iv: "3c1888c1837faee5b405f49687f42098"
server_key_version: "encryptionkey@0"
I20260619 11:13:52.688987 28655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:52.693600 28655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:52.696301 28655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:52.696502 28655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:52.703424 28655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42767
I20260619 11:13:52.703506 28722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42767 every 8 connection(s)
I20260619 11:13:52.704419 28655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:52.707005 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:52.711983 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28655
I20260619 11:13:52.712023 28723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f: Bootstrap starting.
I20260619 11:13:52.712142 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:52.712425 7529 external_mini_cluster.cc:1428] Setting key 44e6e74dada65de0c066aff9d561d491
I20260619 11:13:52.714586 28723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:52.715302 28723 log.cc:824] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:52.717317 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:52.713707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:57525 (local address 127.0.0.1:42767)
0619 11:13:52.714122 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:52.714136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:52.714170 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:13:52.714855 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:52.714873 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:52.715158 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:52.715500 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:52.716184 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:52.716188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:52.716200 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:52.716208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:52.716250 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:52.716267 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:52.716402 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:52.716518 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 11:13:52.716675 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260619 11:13:52.717474 28723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f: No bootstrap required, opened a new log
I20260619 11:13:52.719929 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:52.721186 28723 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260619 11:13:52.721432 28723 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:52.721520 28723 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bdeab7d5c54d92a931db08ddd7c13f, State: Initialized, Role: FOLLOWER
I20260619 11:13:52.722038 28723 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260619 11:13:52.722210 28723 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:52.722275 28723 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:52.722363 28723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:52.723194 28723 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260619 11:13:52.723531 28723 leader_election.cc:302] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0bdeab7d5c54d92a931db08ddd7c13f; no voters:
I20260619 11:13:52.723812 28723 leader_election.cc:288] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:52.723884 28733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:52.724157 28733 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [term 1 LEADER]: Becoming Leader. State: Replica: e0bdeab7d5c54d92a931db08ddd7c13f, State: Running, Role: LEADER
I20260619 11:13:52.724455 28733 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } }
I20260619 11:13:52.725003 28723 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:52.726055 28735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0bdeab7d5c54d92a931db08ddd7c13f. Latest consensus state: current_term: 1 leader_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } } }
I20260619 11:13:52.726183 28735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:52.726431 28734 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bdeab7d5c54d92a931db08ddd7c13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42767 } } }
I20260619 11:13:52.726521 28734 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:52.726634 28742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:52.728597 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:52.721995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:48693 (local address 127.0.0.1:42767)
0619 11:13:52.722111 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:52.722114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:52.727252 (+ 5138us) server_negotiation.cc:407] Connection header received
0619 11:13:52.727379 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:52.727381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:52.727413 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:52.727483 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:52.728189 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:52.728192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:52.728194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:52.728204 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:52.728229 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:52.728246 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:52.728334 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:52.728430 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 11:13:52.728466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:13:52.729629 28742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:52.734524 28742 catalog_manager.cc:1347] Generated new cluster ID: b364df76182a4012a1b843a097f16a89
I20260619 11:13:52.734581 28742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:52.748574 28742 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:52.748844 28742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:52.754021 28742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0bdeab7d5c54d92a931db08ddd7c13f: Generated new TSK 0
I20260619 11:13:52.754557 28742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:52.760576 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28655
2026-06-19T11:13:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:52 dist-test-slave-f1t5 krb5kdc[28754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:52 dist-test-slave-f1t5 krb5kdc[28754](info): set up 2 sockets
Jun 19 11:13:52 dist-test-slave-f1t5 krb5kdc[28754](info): commencing operation
krb5kdc: starting...
W20260619 11:13:53.801745 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:53 dist-test-slave-f1t5 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:53.916430 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:54.019779 28767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:54.020059 28767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:54.020136 28767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:54.024001 28767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:54.024080 28767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:54.024117 28767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:54.024152 28767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:54.024187 28767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:54.029060 28767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:54.029966 28767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:54.030880 28767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:54.038822 28772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:54.038867 28767 server_base.cc:1034] running on GCE node
W20260619 11:13:54.038806 28773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:54.038837 28775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:54.039597 28767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:54.040540 28767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:54.041684 28767 hybrid_clock.cc:648] HybridClock initialized: now 1781867634041669 us; error 29 us; skew 500 ppm
Jun 19 11:13:54 dist-test-slave-f1t5 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:54.044180 28767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:54.044996 28767 webserver.cc:460] Webserver started at http://127.0.0.1:42067/ using document root <none> and password file <none>
I20260619 11:13:54.045466 28767 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:54.045575 28767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:54.045746 28767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:54.047610 28767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "1a25e608a442475f9c4e2286a36c8d7c"
format_stamp: "Formatted at 2026-06-19 11:13:54 on dist-test-slave-f1t5"
server_key: "813add690610fee3c50b34905f1c4880"
server_key_iv: "a8db7d1a40121f5633d3863b29466489"
server_key_version: "encryptionkey@0"
I20260619 11:13:54.048106 28767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "1a25e608a442475f9c4e2286a36c8d7c"
format_stamp: "Formatted at 2026-06-19 11:13:54 on dist-test-slave-f1t5"
server_key: "813add690610fee3c50b34905f1c4880"
server_key_iv: "a8db7d1a40121f5633d3863b29466489"
server_key_version: "encryptionkey@0"
I20260619 11:13:54.051560 28767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:54.053797 28782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:54.054992 28767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:54.055119 28767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "1a25e608a442475f9c4e2286a36c8d7c"
format_stamp: "Formatted at 2026-06-19 11:13:54 on dist-test-slave-f1t5"
server_key: "813add690610fee3c50b34905f1c4880"
server_key_iv: "a8db7d1a40121f5633d3863b29466489"
server_key_version: "encryptionkey@0"
I20260619 11:13:54.055213 28767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:54.069917 28767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:54.072702 28767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:54.072898 28767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:54.079838 28767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41705
I20260619 11:13:54.079864 28834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41705 every 8 connection(s)
I20260619 11:13:54.080724 28767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:54.081075 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28767
I20260619 11:13:54.081208 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:54.081498 7529 external_mini_cluster.cc:1428] Setting key ab10f7432c3ad4c9ef211eba753662aa
I20260619 11:13:54.083729 28835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:54.090030 28835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c: Bootstrap starting.
Jun 19 11:13:54 dist-test-slave-f1t5 krb5kdc[28754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:13:54.092435 28835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:54.093261 28835 log.cc:824] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:54.095507 28835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c: No bootstrap required, opened a new log
I20260619 11:13:54.096236 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:54.082880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:32975 (local address 127.0.0.1:41705)
0619 11:13:54.083293 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:54.083310 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:54.083351 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:13:54.084162 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:54.084183 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:54.084590 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:54.084950 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:54.085620 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:54.087107 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:54.087924 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:54.088586 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:54.088745 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:54.091125 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:54.091130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:54.091143 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:54.091171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:54.093002 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:54.093669 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:54.093676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:54.093684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:54.093790 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:54.094173 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:54.094179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:54.094184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:54.094616 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:54.094810 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:54.095135 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:13:54.095352 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260619 11:13:54.099138 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:54.099596 28835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260619 11:13:54.099792 28835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:54.099833 28835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25e608a442475f9c4e2286a36c8d7c, State: Initialized, Role: FOLLOWER
I20260619 11:13:54.100286 28835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260619 11:13:54.100469 28835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:54.100539 28835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:54.100636 28835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:54.101733 28835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260619 11:13:54.102072 28835 leader_election.cc:302] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a25e608a442475f9c4e2286a36c8d7c; no voters:
I20260619 11:13:54.102401 28835 leader_election.cc:288] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:54.102499 28845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:54.102708 28845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [term 1 LEADER]: Becoming Leader. State: Replica: 1a25e608a442475f9c4e2286a36c8d7c, State: Running, Role: LEADER
I20260619 11:13:54.103053 28845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } }
I20260619 11:13:54.103579 28835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:54.104182 28845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a25e608a442475f9c4e2286a36c8d7c. Latest consensus state: current_term: 1 leader_uuid: "1a25e608a442475f9c4e2286a36c8d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260619 11:13:54.104347 28845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:54.104597 28846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a25e608a442475f9c4e2286a36c8d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25e608a442475f9c4e2286a36c8d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41705 } } }
I20260619 11:13:54.104753 28846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:54.104765 28852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:54.108027 28852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:54.109651 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:54.101130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:49475 (local address 127.0.0.1:41705)
0619 11:13:54.101237 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:54.101240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:54.101385 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:13:54.101451 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:54.101453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:54.101496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:54.101601 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:54.102150 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:54.102941 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:54.104021 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:54.104343 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:54.104443 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:54.106175 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:54.106178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:54.106181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:54.106203 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:54.107814 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:54.108343 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:54.108347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:54.108350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:54.108434 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:54.108838 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:54.108840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:54.108841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:54.108985 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:54.109136 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:54.109397 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 11:13:54.109481 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:13:54.117007 28852 catalog_manager.cc:1347] Generated new cluster ID: f6cadc1006674b93a7c313a27fe10d74
I20260619 11:13:54.117075 28852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:54.130283 28852 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:54.131281 28852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:54.136408 28852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a25e608a442475f9c4e2286a36c8d7c: Generated new TSK 0
I20260619 11:13:54.137012 28852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:54.144933 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28767
W20260619 11:13:54.150384 28752 connection.cc:535] client connection to 127.0.0.1:41705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41705): Connection reset by peer (error 104)
2026-06-19T11:13:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:54 dist-test-slave-f1t5 krb5kdc[28866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:54 dist-test-slave-f1t5 krb5kdc[28866](info): set up 2 sockets
Jun 19 11:13:54 dist-test-slave-f1t5 krb5kdc[28866](info): commencing operation
krb5kdc: starting...
W20260619 11:13:55.161872 7529 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:55 dist-test-slave-f1t5 krb5kdc[28866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:55.276347 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:55.379748 28879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:55.380012 28879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:55.380087 28879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:55.383805 28879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:55.383881 28879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:55.383919 28879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:55.383971 28879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:55.384006 28879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:55.388947 28879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:55.389863 28879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:55.390779 28879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:13:55.398761 28879 server_base.cc:1034] running on GCE node
W20260619 11:13:55.398728 28885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:55.398715 28884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:55.398715 28887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:55.399428 28879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:55.400314 28879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:55.401460 28879 hybrid_clock.cc:648] HybridClock initialized: now 1781867635401443 us; error 28 us; skew 500 ppm
Jun 19 11:13:55 dist-test-slave-f1t5 krb5kdc[28866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:55.404259 28879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:55.405336 28879 webserver.cc:460] Webserver started at http://127.0.0.1:37797/ using document root <none> and password file <none>
I20260619 11:13:55.406116 28879 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:55.406215 28879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:55.406466 28879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:55.408414 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "326bb6c2a55f45949dfbfe2402e65c52"
format_stamp: "Formatted at 2026-06-19 11:13:55 on dist-test-slave-f1t5"
server_key: "237db47d993538e288a4dc47ccab82ca"
server_key_iv: "49cf873b1f5b5653c83c7eaba67f9075"
server_key_version: "encryptionkey@0"
I20260619 11:13:55.408948 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "326bb6c2a55f45949dfbfe2402e65c52"
format_stamp: "Formatted at 2026-06-19 11:13:55 on dist-test-slave-f1t5"
server_key: "237db47d993538e288a4dc47ccab82ca"
server_key_iv: "49cf873b1f5b5653c83c7eaba67f9075"
server_key_version: "encryptionkey@0"
I20260619 11:13:55.412492 28879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:13:55.414710 28894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:55.415868 28879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:13:55.416000 28879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "326bb6c2a55f45949dfbfe2402e65c52"
format_stamp: "Formatted at 2026-06-19 11:13:55 on dist-test-slave-f1t5"
server_key: "237db47d993538e288a4dc47ccab82ca"
server_key_iv: "49cf873b1f5b5653c83c7eaba67f9075"
server_key_version: "encryptionkey@0"
I20260619 11:13:55.416102 28879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:55.433286 28879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:55.435935 28879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:55.436123 28879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:55.442675 28879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37299
I20260619 11:13:55.442701 28946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37299 every 8 connection(s)
I20260619 11:13:55.443516 28879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:55.445976 28947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:55.451054 28947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52: Bootstrap starting.
I20260619 11:13:55.451997 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28879
I20260619 11:13:55.452126 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:55.452392 7529 external_mini_cluster.cc:1428] Setting key 09579e57b31f12c8a28ef66de681a8e0
I20260619 11:13:55.453737 28947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:55.454599 28947 log.cc:824] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:55.456591 28947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52: No bootstrap required, opened a new log
I20260619 11:13:55.460918 28947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } }
I20260619 11:13:55.461140 28947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:55.461194 28947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326bb6c2a55f45949dfbfe2402e65c52, State: Initialized, Role: FOLLOWER
I20260619 11:13:55.461699 28947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } }
I20260619 11:13:55.461860 28947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:55.461988 28947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:55.462081 28947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:55.462191 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:55.453455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:36213 (local address 127.0.0.1:37299)
0619 11:13:55.453961 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:55.453981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:55.454026 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:13:55.454826 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:55.454849 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:55.455201 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:55.455560 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:55.455945 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:55.457370 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:55.458254 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:55.458989 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:55.459179 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:55.460116 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:55.460124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:55.460141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:55.460714 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:55.460773 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:55.460796 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:55.460933 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:55.461119 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 11:13:55.461252 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":140,"threads_started":1}
I20260619 11:13:55.463131 28947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } }
I20260619 11:13:55.463482 28947 leader_election.cc:302] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 326bb6c2a55f45949dfbfe2402e65c52; no voters:
I20260619 11:13:55.463801 28947 leader_election.cc:288] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:55.463869 28952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:55.464064 28952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [term 1 LEADER]: Becoming Leader. State: Replica: 326bb6c2a55f45949dfbfe2402e65c52, State: Running, Role: LEADER
I20260619 11:13:55.464359 28952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } }
I20260619 11:13:55.465025 28947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:55.465116 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:55.465608 28954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 326bb6c2a55f45949dfbfe2402e65c52. Latest consensus state: current_term: 1 leader_uuid: "326bb6c2a55f45949dfbfe2402e65c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } } }
I20260619 11:13:55.465802 28954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:55.466259 28960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:55.467762 28953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "326bb6c2a55f45949dfbfe2402e65c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326bb6c2a55f45949dfbfe2402e65c52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37299 } } }
I20260619 11:13:55.467873 28953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:55.470527 28960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:55.472319 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:55.467467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:57539 (local address 127.0.0.1:37299)
0619 11:13:55.467568 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:55.467572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:55.467833 (+ 261us) server_negotiation.cc:407] Connection header received
0619 11:13:55.468004 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:55.468007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:55.468050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:55.468139 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:55.468388 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:55.469074 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:55.470274 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:55.470583 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:55.470678 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:55.471700 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:55.471704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:55.471707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:55.471719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:55.471746 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:55.471766 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:55.471845 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:55.472161 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 11:13:55.472192 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:13:55.475782 28960 catalog_manager.cc:1347] Generated new cluster ID: f6b6b4d8fa1c499abb7dee2aa668fe86
I20260619 11:13:55.475841 28960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:55.483162 28960 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:55.483983 28960 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:55.488765 28960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 326bb6c2a55f45949dfbfe2402e65c52: Generated new TSK 0
I20260619 11:13:55.489364 28960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:55.492939 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28879
W20260619 11:13:55.498432 28864 connection.cc:535] client connection to 127.0.0.1:37299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37299): Connection reset by peer (error 104)
2026-06-19T11:13:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:55 dist-test-slave-f1t5 krb5kdc[28978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:55 dist-test-slave-f1t5 krb5kdc[28978](info): set up 2 sockets
Jun 19 11:13:55 dist-test-slave-f1t5 krb5kdc[28978](info): commencing operation
krb5kdc: starting...
W20260619 11:13:56.529747 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:56 dist-test-slave-f1t5 krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:13:56.643857 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:56.747247 28991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:56.747551 28991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:56.747627 28991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:56.751809 28991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:56.751897 28991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:56.751965 28991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:56.752029 28991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:56.752064 28991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:56.757299 28991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:56.758315 28991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:56.759294 28991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:56.767416 28999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:56.767416 28997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:56.767432 28996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:56.767746 28991 server_base.cc:1034] running on GCE node
I20260619 11:13:56.768321 28991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:56.769280 28991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:56.770452 28991 hybrid_clock.cc:648] HybridClock initialized: now 1781867636770431 us; error 42 us; skew 500 ppm
Jun 19 11:13:56 dist-test-slave-f1t5 krb5kdc[28978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:56.776095 28991 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:13:56.776922 28991 webserver.cc:460] Webserver started at http://127.0.0.1:36549/ using document root <none> and password file <none>
I20260619 11:13:56.777433 28991 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:56.777551 28991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:56.777768 28991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:56.779660 28991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "c921b5ad041742a1865fd09b41ad84d5"
format_stamp: "Formatted at 2026-06-19 11:13:56 on dist-test-slave-f1t5"
server_key: "fdb5a19537a275ccf52d294693914419"
server_key_iv: "44cc66d05312d01886efba64e5c1ad79"
server_key_version: "encryptionkey@0"
I20260619 11:13:56.780189 28991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "c921b5ad041742a1865fd09b41ad84d5"
format_stamp: "Formatted at 2026-06-19 11:13:56 on dist-test-slave-f1t5"
server_key: "fdb5a19537a275ccf52d294693914419"
server_key_iv: "44cc66d05312d01886efba64e5c1ad79"
server_key_version: "encryptionkey@0"
I20260619 11:13:56.783650 28991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:56.785849 29006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:56.787004 28991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:13:56.787133 28991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "c921b5ad041742a1865fd09b41ad84d5"
format_stamp: "Formatted at 2026-06-19 11:13:56 on dist-test-slave-f1t5"
server_key: "fdb5a19537a275ccf52d294693914419"
server_key_iv: "44cc66d05312d01886efba64e5c1ad79"
server_key_version: "encryptionkey@0"
I20260619 11:13:56.787232 28991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:56.791424 28991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:56.794279 28991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:56.794454 28991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:56.801084 28991 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41143
I20260619 11:13:56.801108 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41143 every 8 connection(s)
I20260619 11:13:56.801957 28991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:56.804397 29059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:56.808925 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 28991
I20260619 11:13:56.809067 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:56.809331 7529 external_mini_cluster.cc:1428] Setting key d79f8bbf1d885fe6df07036cb9bb6e33
I20260619 11:13:56.809719 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5: Bootstrap starting.
I20260619 11:13:56.811983 29059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:56.812796 29059 log.cc:824] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:56.814220 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:56.810620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:47661 (local address 127.0.0.1:41143)
0619 11:13:56.811061 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:56.811075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:56.811106 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:13:56.811676 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:56.811692 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:56.811972 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:56.812321 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:56.813002 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:56.813008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:56.813022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:56.813032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:56.813085 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:56.813123 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:56.813246 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:56.813391 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 11:13:56.813582 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260619 11:13:56.814718 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5: No bootstrap required, opened a new log
I20260619 11:13:56.816785 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:56.818296 29059 raft_consensus.cc:348] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260619 11:13:56.818528 29059 raft_consensus.cc:374] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:56.818603 29059 raft_consensus.cc:729] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c921b5ad041742a1865fd09b41ad84d5, State: Initialized, Role: FOLLOWER
I20260619 11:13:56.819061 29059 consensus_queue.cc:260] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260619 11:13:56.819252 29059 raft_consensus.cc:388] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:56.819325 29059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:56.819437 29059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:56.820379 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:56.818775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:43173 (local address 127.0.0.1:41143)
0619 11:13:56.818899 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:56.818903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:56.819066 (+ 163us) server_negotiation.cc:407] Connection header received
0619 11:13:56.819176 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:56.819177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:56.819227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:56.819321 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:56.820017 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:56.820020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:56.820022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:56.820033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:56.820057 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:56.820070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:56.820163 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:56.820232 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 11:13:56.820271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:13:56.820427 29059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260619 11:13:56.820779 29059 leader_election.cc:302] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c921b5ad041742a1865fd09b41ad84d5; no voters:
I20260619 11:13:56.821085 29059 leader_election.cc:288] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:56.821177 29069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:56.821370 29069 raft_consensus.cc:686] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [term 1 LEADER]: Becoming Leader. State: Replica: c921b5ad041742a1865fd09b41ad84d5, State: Running, Role: LEADER
I20260619 11:13:56.821717 29069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260619 11:13:56.822223 29059 sys_catalog.cc:564] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:56.823230 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c921b5ad041742a1865fd09b41ad84d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260619 11:13:56.823364 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:56.823742 29078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:56.823711 29071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c921b5ad041742a1865fd09b41ad84d5. Latest consensus state: current_term: 1 leader_uuid: "c921b5ad041742a1865fd09b41ad84d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c921b5ad041742a1865fd09b41ad84d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260619 11:13:56.823839 29071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:56.826337 29078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:56.831075 29078 catalog_manager.cc:1347] Generated new cluster ID: decca76ef4594eb8ac450bd5bff0d5f4
I20260619 11:13:56.831133 29078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:56.851388 29078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:56.851691 29078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:56.864734 29078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c921b5ad041742a1865fd09b41ad84d5: Generated new TSK 0
I20260619 11:13:56.865477 29078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:56.869261 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 28991
2026-06-19T11:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:56 dist-test-slave-f1t5 krb5kdc[29090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:56 dist-test-slave-f1t5 krb5kdc[29090](info): set up 2 sockets
Jun 19 11:13:56 dist-test-slave-f1t5 krb5kdc[29090](info): commencing operation
krb5kdc: starting...
W20260619 11:13:57.889941 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:57 dist-test-slave-f1t5 krb5kdc[29090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:13:58.004164 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35293
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:58.108115 29103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:58.108372 29103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:58.108458 29103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:58.112334 29103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:58.112409 29103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:58.112445 29103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:58.112478 29103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:58.112505 29103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:58.117470 29103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:58.118376 29103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:58.119259 29103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:13:58.127070 29103 server_base.cc:1034] running on GCE node
W20260619 11:13:58.127032 29111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:58.127032 29109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:58.127115 29108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:58.127784 29103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:58.128741 29103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:58.129935 29103 hybrid_clock.cc:648] HybridClock initialized: now 1781867638129911 us; error 35 us; skew 500 ppm
Jun 19 11:13:58 dist-test-slave-f1t5 krb5kdc[29090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867638, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:58.132241 29103 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:58.132930 29103 webserver.cc:460] Webserver started at http://127.7.90.126:46447/ using document root <none> and password file <none>
I20260619 11:13:58.133350 29103 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:58.133409 29103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:58.133641 29103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:58.135432 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "0a2e44eb9ed44e66a0ae153f35975a02"
format_stamp: "Formatted at 2026-06-19 11:13:58 on dist-test-slave-f1t5"
server_key: "6ad3b0615b227a4d714a1e598dfbeecc"
server_key_iv: "58f2f4148342717f59053de81d10803b"
server_key_version: "encryptionkey@0"
I20260619 11:13:58.135914 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "0a2e44eb9ed44e66a0ae153f35975a02"
format_stamp: "Formatted at 2026-06-19 11:13:58 on dist-test-slave-f1t5"
server_key: "6ad3b0615b227a4d714a1e598dfbeecc"
server_key_iv: "58f2f4148342717f59053de81d10803b"
server_key_version: "encryptionkey@0"
I20260619 11:13:58.139410 29103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:13:58.141487 29118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:58.142683 29103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:58.142782 29103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "0a2e44eb9ed44e66a0ae153f35975a02"
format_stamp: "Formatted at 2026-06-19 11:13:58 on dist-test-slave-f1t5"
server_key: "6ad3b0615b227a4d714a1e598dfbeecc"
server_key_iv: "58f2f4148342717f59053de81d10803b"
server_key_version: "encryptionkey@0"
I20260619 11:13:58.142911 29103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:58.161953 29103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:58.164497 29103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:58.164676 29103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:58.171242 29103 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:35293
I20260619 11:13:58.171257 29170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:35293 every 8 connection(s)
I20260619 11:13:58.172195 29103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:58.174746 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:58.179636 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02: Bootstrap starting.
I20260619 11:13:58.179757 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29103
I20260619 11:13:58.179967 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:58.180217 7529 external_mini_cluster.cc:1428] Setting key 40f99a4b710850675b603473a7d1c4e6
I20260619 11:13:58.182160 29171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:58.182868 29171 log.cc:824] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:58.184566 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02: No bootstrap required, opened a new log
I20260619 11:13:58.187943 29171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } }
I20260619 11:13:58.188180 29171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:58.188259 29171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2e44eb9ed44e66a0ae153f35975a02, State: Initialized, Role: FOLLOWER
Jun 19 11:13:58 dist-test-slave-f1t5 krb5kdc[29090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:13:58.188692 29171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } }
I20260619 11:13:58.188876 29171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:58.188963 29171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:58.189076 29171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:58.190166 29171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } }
I20260619 11:13:58.190550 29171 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2e44eb9ed44e66a0ae153f35975a02; no voters:
I20260619 11:13:58.190861 29171 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:58.190963 29176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:58.191152 29176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2e44eb9ed44e66a0ae153f35975a02, State: Running, Role: LEADER
I20260619 11:13:58.191462 29176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } }
I20260619 11:13:58.192106 29171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:58.193045 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } } }
I20260619 11:13:58.193167 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:58.193569 29178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2e44eb9ed44e66a0ae153f35975a02. Latest consensus state: current_term: 1 leader_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e44eb9ed44e66a0ae153f35975a02" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35293 } } }
I20260619 11:13:58.193872 29178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02 [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:58.194281 29185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:58.195247 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:58.181413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47623 (local address 127.7.90.126:35293)
0619 11:13:58.181825 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:58.181840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:58.181872 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:13:58.182544 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:58.182560 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:58.182877 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:58.183239 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:58.184020 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:58.185004 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:58.185915 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:58.186363 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:58.186529 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:58.189171 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:58.189176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:58.189191 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:58.189220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:58.191142 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:58.191760 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:58.191766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:58.191771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:58.191903 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:58.192323 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:58.192326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:58.192328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:58.192924 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:58.193797 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:58.193995 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 11:13:58.194225 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260619 11:13:58.197759 29185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:58.198300 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:58.203161 29185 catalog_manager.cc:1347] Generated new cluster ID: 347f901de9d0484888eb35f12a438d91
I20260619 11:13:58.203212 29185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:58.210369 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:58.200287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44773 (local address 127.7.90.126:35293)
0619 11:13:58.200394 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:58.200398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:58.201815 (+ 1417us) server_negotiation.cc:407] Connection header received
0619 11:13:58.201935 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:58.201938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:58.201982 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:58.202065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:58.202811 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:58.203961 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:58.205009 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:58.205381 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:58.205498 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:58.207007 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:58.207011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:58.207014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:13:58.207038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:58.208639 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:58.209102 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:58.209105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:58.209107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:58.209182 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:13:58.209468 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:13:58.209471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:13:58.209472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:13:58.209674 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:58.209829 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:58.210108 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 11:13:58.210196 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:13:58.219123 29185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:58.220249 29185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:58.229099 29185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2e44eb9ed44e66a0ae153f35975a02: Generated new TSK 0
I20260619 11:13:58.229801 29185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:58.240725 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29103
W20260619 11:13:58.245988 29088 connection.cc:535] client connection to 127.7.90.126:35293 recv error: Network error: failed to read from TLS socket (remote: 127.7.90.126:35293): Connection reset by peer (error 104)
2026-06-19T11:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:58 dist-test-slave-f1t5 krb5kdc[29202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:58 dist-test-slave-f1t5 krb5kdc[29202](info): set up 2 sockets
Jun 19 11:13:58 dist-test-slave-f1t5 krb5kdc[29202](info): commencing operation
krb5kdc: starting...
W20260619 11:13:59.273865 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:13:59 dist-test-slave-f1t5 krb5kdc[29202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:13:59.388026 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38593
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:13:59.493023 29215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:13:59.493235 29215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:13:59.493273 29215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:13:59.497067 29215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:13:59.497116 29215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:13:59.497134 29215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:13:59.497150 29215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:13:59.497165 29215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:13:59.502127 29215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:38593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:13:59.503053 29215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:13:59.503901 29215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:13:59.512008 29223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:13:59.511981 29220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:59.512063 29215 server_base.cc:1034] running on GCE node
W20260619 11:13:59.511993 29221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:13:59.512810 29215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:13:59.513921 29215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:13:59.515077 29215 hybrid_clock.cc:648] HybridClock initialized: now 1781867639515061 us; error 30 us; skew 500 ppm
Jun 19 11:13:59 dist-test-slave-f1t5 krb5kdc[29202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867639, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:13:59.517645 29215 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:13:59.518455 29215 webserver.cc:460] Webserver started at http://127.7.90.126:43159/ using document root <none> and password file <none>
I20260619 11:13:59.519017 29215 fs_manager.cc:359] Metadata directory not provided
I20260619 11:13:59.519107 29215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:13:59.519327 29215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:13:59.521311 29215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "ee22b96b146940e9be23158bd042bc2c"
format_stamp: "Formatted at 2026-06-19 11:13:59 on dist-test-slave-f1t5"
server_key: "bf5d71c23de1df2b1dfea464fedeb92f"
server_key_iv: "aad48c155d23d20bafb1beed74531d82"
server_key_version: "encryptionkey@0"
I20260619 11:13:59.521916 29215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "ee22b96b146940e9be23158bd042bc2c"
format_stamp: "Formatted at 2026-06-19 11:13:59 on dist-test-slave-f1t5"
server_key: "bf5d71c23de1df2b1dfea464fedeb92f"
server_key_iv: "aad48c155d23d20bafb1beed74531d82"
server_key_version: "encryptionkey@0"
I20260619 11:13:59.525318 29215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:13:59.527552 29230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:13:59.528767 29215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:13:59.528908 29215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "ee22b96b146940e9be23158bd042bc2c"
format_stamp: "Formatted at 2026-06-19 11:13:59 on dist-test-slave-f1t5"
server_key: "bf5d71c23de1df2b1dfea464fedeb92f"
server_key_iv: "aad48c155d23d20bafb1beed74531d82"
server_key_version: "encryptionkey@0"
I20260619 11:13:59.529024 29215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:13:59.542537 29215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:13:59.545145 29215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:13:59.545332 29215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:13:59.552136 29215 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:38593
I20260619 11:13:59.552160 29282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:38593 every 8 connection(s)
I20260619 11:13:59.552990 29215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:13:59.553280 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29215
I20260619 11:13:59.553450 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:13:59.553754 7529 external_mini_cluster.cc:1428] Setting key 95775be817cbf50137d48e4ed4f49305
I20260619 11:13:59.555994 29283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:13:59.562171 29283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c: Bootstrap starting.
I20260619 11:13:59.562500 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:59.555054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58701 (local address 127.7.90.126:38593)
0619 11:13:59.555482 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:59.555502 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:59.555533 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:13:59.556196 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:59.556213 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:59.556529 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:59.556783 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:59.557151 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:59.558179 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:59.559379 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:59.559923 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:59.560031 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:59.560833 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:59.560838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:59.560849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:59.561207 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:59.561253 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:59.561273 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:59.561375 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:59.561667 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 11:13:59.561801 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":131,"threads_started":1}
I20260619 11:13:59.564457 29283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c: Neither blocks nor log segments found. Creating new log.
I20260619 11:13:59.565258 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:13:59.565271 29283 log.cc:824] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:13:59.567139 29283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c: No bootstrap required, opened a new log
I20260619 11:13:59.570837 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:13:59.567282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48903 (local address 127.7.90.126:38593)
0619 11:13:59.567393 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:13:59.567397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:13:59.567558 (+ 161us) server_negotiation.cc:407] Connection header received
0619 11:13:59.567638 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:13:59.567641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:13:59.567679 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:13:59.567750 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:13:59.567956 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:59.568646 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:59.569347 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:13:59.569677 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:13:59.569780 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:13:59.570481 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:13:59.570485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:13:59.570488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:13:59.570499 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:13:59.570523 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:13:59.570536 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:13:59.570628 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:13:59.570695 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 11:13:59.570726 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:13:59.571156 29283 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } }
I20260619 11:13:59.571348 29283 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:13:59.571406 29283 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee22b96b146940e9be23158bd042bc2c, State: Initialized, Role: FOLLOWER
I20260619 11:13:59.571808 29283 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } }
I20260619 11:13:59.571959 29283 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:13:59.572018 29283 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:13:59.572103 29283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:13:59.572994 29283 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } }
I20260619 11:13:59.573292 29283 leader_election.cc:302] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee22b96b146940e9be23158bd042bc2c; no voters:
I20260619 11:13:59.573633 29283 leader_election.cc:288] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:13:59.573694 29293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:13:59.573918 29293 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [term 1 LEADER]: Becoming Leader. State: Replica: ee22b96b146940e9be23158bd042bc2c, State: Running, Role: LEADER
I20260619 11:13:59.574224 29293 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } }
I20260619 11:13:59.574776 29283 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:13:59.575809 29294 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee22b96b146940e9be23158bd042bc2c. Latest consensus state: current_term: 1 leader_uuid: "ee22b96b146940e9be23158bd042bc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } } }
I20260619 11:13:59.575932 29294 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:59.576289 29302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:13:59.576248 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee22b96b146940e9be23158bd042bc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee22b96b146940e9be23158bd042bc2c" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38593 } } }
I20260619 11:13:59.576818 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c [sys.catalog]: This master's current role is: LEADER
I20260619 11:13:59.579550 29302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:13:59.584640 29302 catalog_manager.cc:1347] Generated new cluster ID: f3332d5350904458a592375c1c939f80
I20260619 11:13:59.584709 29302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:13:59.591105 29302 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:13:59.591837 29302 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:13:59.609081 29302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee22b96b146940e9be23158bd042bc2c: Generated new TSK 0
I20260619 11:13:59.609866 29302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:13:59.623982 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29215
W20260619 11:13:59.629309 29200 connection.cc:535] client connection to 127.7.90.126:38593 recv error: Network error: failed to read from TLS socket (remote: 127.7.90.126:38593): Connection reset by peer (error 104)
2026-06-19T11:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:13:59 dist-test-slave-f1t5 krb5kdc[29314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:13:59 dist-test-slave-f1t5 krb5kdc[29314](info): set up 2 sockets
Jun 19 11:13:59 dist-test-slave-f1t5 krb5kdc[29314](info): commencing operation
krb5kdc: starting...
W20260619 11:14:00.657761 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:00 dist-test-slave-f1t5 krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:00.774114 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36857
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:00.882754 29327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:00.882995 29327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:00.883034 29327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:00.886931 29327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:00.886984 29327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:00.887003 29327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:00.887019 29327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:00.887034 29327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:00.891986 29327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:36857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:00.892891 29327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:00.893939 29327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:00.902733 29333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:00.902799 29327 server_base.cc:1034] running on GCE node
W20260619 11:14:00.902757 29335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:00.902733 29332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:00.903553 29327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:00.904592 29327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:00.905772 29327 hybrid_clock.cc:648] HybridClock initialized: now 1781867640905761 us; error 30 us; skew 500 ppm
Jun 19 11:14:00 dist-test-slave-f1t5 krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867640, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:00.908476 29327 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:00.909376 29327 webserver.cc:460] Webserver started at http://127.7.90.126:45747/ using document root <none> and password file <none>
I20260619 11:14:00.909970 29327 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:00.910058 29327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:00.910269 29327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:00.912266 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f"
format_stamp: "Formatted at 2026-06-19 11:14:00 on dist-test-slave-f1t5"
server_key: "e459d304092e31cc364fd3bc64462b70"
server_key_iv: "28532d83a2d6a429c2da9f44e6223c6f"
server_key_version: "encryptionkey@0"
I20260619 11:14:00.912823 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f"
format_stamp: "Formatted at 2026-06-19 11:14:00 on dist-test-slave-f1t5"
server_key: "e459d304092e31cc364fd3bc64462b70"
server_key_iv: "28532d83a2d6a429c2da9f44e6223c6f"
server_key_version: "encryptionkey@0"
I20260619 11:14:00.916466 29327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:14:00.918851 29342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:00.920027 29327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:14:00.920164 29327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f"
format_stamp: "Formatted at 2026-06-19 11:14:00 on dist-test-slave-f1t5"
server_key: "e459d304092e31cc364fd3bc64462b70"
server_key_iv: "28532d83a2d6a429c2da9f44e6223c6f"
server_key_version: "encryptionkey@0"
I20260619 11:14:00.920272 29327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:00.924768 29327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:00.927920 29327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:00.928109 29327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:00.934656 29327 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:36857
I20260619 11:14:00.934721 29394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:36857 every 8 connection(s)
I20260619 11:14:00.935552 29327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:00.938216 29395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:00.939921 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29327
I20260619 11:14:00.940083 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:00.940368 7529 external_mini_cluster.cc:1428] Setting key ce73f92e23041be61c65f9964e6c015a
I20260619 11:14:00.944901 29395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f: Bootstrap starting.
I20260619 11:14:00.945731 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:00.941750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43097 (local address 127.7.90.126:36857)
0619 11:14:00.942183 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:00.942202 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:00.942254 (+ 52us) server_negotiation.cc:407] Connection header received
0619 11:14:00.943093 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:00.943116 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:00.943457 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:00.943821 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:00.944431 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:00.944436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:00.944450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:00.944460 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:00.944515 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:00.944540 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:00.944665 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:00.944728 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:14:00.944866 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260619 11:14:00.947439 29395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:00.948191 29395 log.cc:824] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:00.948486 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:00.950423 29395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f: No bootstrap required, opened a new log
I20260619 11:14:00.952098 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:00.950626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44259 (local address 127.7.90.126:36857)
0619 11:14:00.950726 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:00.950730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:00.950922 (+ 192us) server_negotiation.cc:407] Connection header received
0619 11:14:00.951006 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:00.951008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:00.951039 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:00.951112 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:00.951766 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:00.951771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:00.951776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:00.951787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:00.951813 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:00.951825 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:00.951923 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:00.951957 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 11:14:00.951990 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:14:00.954609 29395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } }
I20260619 11:14:00.954804 29395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:14:00.954835 29395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae6a0e07a9945e3b9def15e5ddc7c6f, State: Initialized, Role: FOLLOWER
I20260619 11:14:00.955163 29395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } }
I20260619 11:14:00.955289 29395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:14:00.955353 29395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:14:00.955457 29395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:00.956449 29395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } }
I20260619 11:14:00.956766 29395 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae6a0e07a9945e3b9def15e5ddc7c6f; no voters:
I20260619 11:14:00.957034 29395 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:14:00.957154 29405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:00.957384 29405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [term 1 LEADER]: Becoming Leader. State: Replica: 9ae6a0e07a9945e3b9def15e5ddc7c6f, State: Running, Role: LEADER
I20260619 11:14:00.957720 29405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } }
I20260619 11:14:00.958204 29395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:00.959338 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae6a0e07a9945e3b9def15e5ddc7c6f. Latest consensus state: current_term: 1 leader_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } } }
I20260619 11:14:00.959471 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:00.959755 29406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6a0e07a9945e3b9def15e5ddc7c6f" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 36857 } } }
I20260619 11:14:00.959847 29406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:00.959847 29414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:00.962750 29414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:00.967733 29414 catalog_manager.cc:1347] Generated new cluster ID: 2bec88a42ade49589b8ccb940951eb61
I20260619 11:14:00.967805 29414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:00.981104 29414 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:00.981418 29414 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:00.991485 29414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae6a0e07a9945e3b9def15e5ddc7c6f: Generated new TSK 0
I20260619 11:14:00.992152 29414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:14:00.995849 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29327
2026-06-19T11:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:14:01 dist-test-slave-f1t5 krb5kdc[29426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:14:01 dist-test-slave-f1t5 krb5kdc[29426](info): set up 2 sockets
Jun 19 11:14:01 dist-test-slave-f1t5 krb5kdc[29426](info): commencing operation
krb5kdc: starting...
W20260619 11:14:02.021766 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:02.136498 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46101
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:02.243602 29439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:02.243837 29439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:02.243875 29439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:02.247851 29439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:02.247898 29439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:02.247916 29439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:02.247931 29439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:02.247944 29439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:02.252831 29439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:02.253814 29439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:02.254752 29439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:02.263016 29444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:02.263069 29439 server_base.cc:1034] running on GCE node
W20260619 11:14:02.263060 29445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:02.263033 29447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:02.263715 29439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:02.264729 29439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:02.265914 29439 hybrid_clock.cc:648] HybridClock initialized: now 1781867642265901 us; error 22 us; skew 500 ppm
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867642, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:02.273207 29439 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:02.274108 29439 webserver.cc:460] Webserver started at http://127.7.90.126:35363/ using document root <none> and password file <none>
I20260619 11:14:02.274611 29439 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:02.274667 29439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:02.274821 29439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:02.276711 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "6d7cf1f4489c4e748081e6152241bd95"
format_stamp: "Formatted at 2026-06-19 11:14:02 on dist-test-slave-f1t5"
server_key: "f99879fee9a5e268de83260f1c38cff4"
server_key_iv: "a1d0ce0a770fd72fcbb83b36f35d9980"
server_key_version: "encryptionkey@0"
I20260619 11:14:02.277194 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "6d7cf1f4489c4e748081e6152241bd95"
format_stamp: "Formatted at 2026-06-19 11:14:02 on dist-test-slave-f1t5"
server_key: "f99879fee9a5e268de83260f1c38cff4"
server_key_iv: "a1d0ce0a770fd72fcbb83b36f35d9980"
server_key_version: "encryptionkey@0"
I20260619 11:14:02.280761 29439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:14:02.283012 29454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:02.284181 29439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:14:02.284312 29439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "6d7cf1f4489c4e748081e6152241bd95"
format_stamp: "Formatted at 2026-06-19 11:14:02 on dist-test-slave-f1t5"
server_key: "f99879fee9a5e268de83260f1c38cff4"
server_key_iv: "a1d0ce0a770fd72fcbb83b36f35d9980"
server_key_version: "encryptionkey@0"
I20260619 11:14:02.284415 29439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:02.298108 29439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:02.300722 29439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:02.300930 29439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:02.307699 29439 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:46101
I20260619 11:14:02.307739 29506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:46101 every 8 connection(s)
I20260619 11:14:02.308581 29439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:02.311285 29507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:02.311946 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29439
I20260619 11:14:02.312088 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:02.312426 7529 external_mini_cluster.cc:1428] Setting key d3b253d4c38fc842f4a90c253612e5de
I20260619 11:14:02.317479 29507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95: Bootstrap starting.
I20260619 11:14:02.320011 29507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:02.320825 29507 log.cc:824] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:02.322817 29507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95: No bootstrap required, opened a new log
I20260619 11:14:02.325875 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:02.313719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33865 (local address 127.7.90.126:46101)
0619 11:14:02.314101 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:02.314127 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:02.314161 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:14:02.314791 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:02.314806 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:02.315122 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:02.315392 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:02.316304 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:02.317239 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:02.318379 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:02.318811 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:02.318926 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:02.321591 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:02.321596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:02.321608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:02.321633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:02.323254 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:02.323684 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:02.323690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:02.323694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:02.323794 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:02.324074 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:02.324078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:02.324079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:02.324431 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:02.324618 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:02.324910 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 11:14:02.325165 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260619 11:14:02.326287 29507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } }
I20260619 11:14:02.326515 29507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:14:02.326604 29507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7cf1f4489c4e748081e6152241bd95, State: Initialized, Role: FOLLOWER
I20260619 11:14:02.327059 29507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } }
I20260619 11:14:02.327204 29507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:14:02.327292 29507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:14:02.327389 29507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:02.328372 29507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } }
I20260619 11:14:02.328747 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:02.328724 29507 leader_election.cc:302] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d7cf1f4489c4e748081e6152241bd95; no voters:
I20260619 11:14:02.329026 29507 leader_election.cc:288] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:14:02.329125 29512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:02.329329 29512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [term 1 LEADER]: Becoming Leader. State: Replica: 6d7cf1f4489c4e748081e6152241bd95, State: Running, Role: LEADER
I20260619 11:14:02.329715 29512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } }
I20260619 11:14:02.330284 29507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:02.330857 29514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d7cf1f4489c4e748081e6152241bd95. Latest consensus state: current_term: 1 leader_uuid: "6d7cf1f4489c4e748081e6152241bd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } } }
I20260619 11:14:02.330997 29514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:02.331337 29520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:02.331284 29513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d7cf1f4489c4e748081e6152241bd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d7cf1f4489c4e748081e6152241bd95" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 46101 } } }
I20260619 11:14:02.331388 29513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:02.334421 29520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:02.340250 29520 catalog_manager.cc:1347] Generated new cluster ID: 28f822495a7846b9893e48f857929457
I20260619 11:14:02.340312 29520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:02.340803 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:02.330877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33589 (local address 127.7.90.126:46101)
0619 11:14:02.330987 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:02.330991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:02.332141 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 11:14:02.332370 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:02.332373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:02.332420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:02.332475 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:02.333406 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:02.334158 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:02.335249 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:02.335560 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:02.335661 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:02.337283 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:02.337287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:02.337290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:02.337318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:02.338944 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:02.339424 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:02.339430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:02.339431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:02.339517 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:02.339812 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:02.339815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:02.339817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:02.340024 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:02.340157 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:02.340392 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 11:14:02.340510 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:14:02.356374 29520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:02.357409 29520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:02.363008 29520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d7cf1f4489c4e748081e6152241bd95: Generated new TSK 0
I20260619 11:14:02.363634 29520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:14:02.367919 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29439
2026-06-19T11:14:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29538](info): set up 2 sockets
Jun 19 11:14:02 dist-test-slave-f1t5 krb5kdc[29538](info): commencing operation
krb5kdc: starting...
W20260619 11:14:03.393797 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:03 dist-test-slave-f1t5 krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:03.509833 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41299
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:03.615552 29551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:03.615832 29551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:03.615913 29551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:03.619750 29551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:03.619829 29551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:03.619866 29551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:03.619922 29551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:03.619968 29551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:03.624691 29551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:41299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:03.625639 29551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:03.626586 29551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:03.634449 29556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:03.634452 29557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:03.634598 29551 server_base.cc:1034] running on GCE node
W20260619 11:14:03.634817 29559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:03.635337 29551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:03.636284 29551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:03.637449 29551 hybrid_clock.cc:648] HybridClock initialized: now 1781867643637436 us; error 31 us; skew 500 ppm
Jun 19 11:14:03 dist-test-slave-f1t5 krb5kdc[29538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867643, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:03.643492 29551 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:03.644294 29551 webserver.cc:460] Webserver started at http://127.7.90.126:42889/ using document root <none> and password file <none>
I20260619 11:14:03.644805 29551 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:03.644866 29551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:03.645092 29551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:03.647027 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "25ae2740a15640f0a9170f0d3e109c41"
format_stamp: "Formatted at 2026-06-19 11:14:03 on dist-test-slave-f1t5"
server_key: "da4fe34d508c5edd9791a563c6307b0d"
server_key_iv: "a2c408eeeaf9dd8ff13ba1732de9af37"
server_key_version: "encryptionkey@0"
I20260619 11:14:03.647562 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "25ae2740a15640f0a9170f0d3e109c41"
format_stamp: "Formatted at 2026-06-19 11:14:03 on dist-test-slave-f1t5"
server_key: "da4fe34d508c5edd9791a563c6307b0d"
server_key_iv: "a2c408eeeaf9dd8ff13ba1732de9af37"
server_key_version: "encryptionkey@0"
I20260619 11:14:03.650846 29551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:14:03.652954 29566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:03.654150 29551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:14:03.654277 29551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "25ae2740a15640f0a9170f0d3e109c41"
format_stamp: "Formatted at 2026-06-19 11:14:03 on dist-test-slave-f1t5"
server_key: "da4fe34d508c5edd9791a563c6307b0d"
server_key_iv: "a2c408eeeaf9dd8ff13ba1732de9af37"
server_key_version: "encryptionkey@0"
I20260619 11:14:03.654386 29551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:03.684270 29551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:03.687148 29551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:03.687347 29551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:03.694056 29551 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:41299
I20260619 11:14:03.694079 29618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:41299 every 8 connection(s)
I20260619 11:14:03.694962 29551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:03.695323 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29551
I20260619 11:14:03.695487 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:03.695788 7529 external_mini_cluster.cc:1428] Setting key f065c9677aa674f7bdbb8f49ec1a5127
I20260619 11:14:03.697751 29619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:03.703816 29619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41: Bootstrap starting.
I20260619 11:14:03.704404 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:03.697141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53967 (local address 127.7.90.126:41299)
0619 11:14:03.697564 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:03.697582 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:03.697626 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:14:03.698276 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:03.698292 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:03.698614 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:03.698875 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:03.699222 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:03.700172 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:03.701385 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:03.701883 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:03.701996 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:03.702767 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:03.702771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:03.702782 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:03.703135 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:03.703184 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:03.703204 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:03.703275 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:03.703449 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 11:14:03.703684 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":131,"threads_started":1}
I20260619 11:14:03.706353 29619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:03.707073 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:03.707238 29619 log.cc:824] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:03.709358 29619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41: No bootstrap required, opened a new log
I20260619 11:14:03.713253 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:03.709213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50485 (local address 127.7.90.126:41299)
0619 11:14:03.709309 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:03.709330 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:03.709495 (+ 165us) server_negotiation.cc:407] Connection header received
0619 11:14:03.709688 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:03.709690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:03.709726 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:03.709787 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:03.710053 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:03.710518 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:03.711621 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:03.711835 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:03.711904 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:03.712768 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:03.712772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:03.712775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:03.712788 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:03.712836 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:03.712851 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:03.712934 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:03.713065 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 11:14:03.713106 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:14:03.713929 29619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } }
I20260619 11:14:03.714157 29619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:14:03.714219 29619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae2740a15640f0a9170f0d3e109c41, State: Initialized, Role: FOLLOWER
I20260619 11:14:03.714632 29619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } }
I20260619 11:14:03.714751 29619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:14:03.714803 29619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:14:03.714871 29619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:03.715629 29619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } }
I20260619 11:14:03.715915 29619 leader_election.cc:302] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ae2740a15640f0a9170f0d3e109c41; no voters:
I20260619 11:14:03.716146 29619 leader_election.cc:288] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:14:03.716230 29629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:03.716511 29629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [term 1 LEADER]: Becoming Leader. State: Replica: 25ae2740a15640f0a9170f0d3e109c41, State: Running, Role: LEADER
I20260619 11:14:03.716920 29629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } }
I20260619 11:14:03.717489 29619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:03.718712 29630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ae2740a15640f0a9170f0d3e109c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } } }
I20260619 11:14:03.718845 29630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:03.719172 29631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ae2740a15640f0a9170f0d3e109c41. Latest consensus state: current_term: 1 leader_uuid: "25ae2740a15640f0a9170f0d3e109c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ae2740a15640f0a9170f0d3e109c41" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 41299 } } }
I20260619 11:14:03.719314 29631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:03.719621 29638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:03.723363 29638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:03.730696 29638 catalog_manager.cc:1347] Generated new cluster ID: 6d1447f107364c6eb5f5efc351aedb95
I20260619 11:14:03.730813 29638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:03.777120 29638 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:03.778203 29638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:03.785780 29638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ae2740a15640f0a9170f0d3e109c41: Generated new TSK 0
I20260619 11:14:03.786504 29638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:14:03.797184 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29551
2026-06-19T11:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:14:03 dist-test-slave-f1t5 krb5kdc[29650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:14:03 dist-test-slave-f1t5 krb5kdc[29650](info): set up 2 sockets
Jun 19 11:14:03 dist-test-slave-f1t5 krb5kdc[29650](info): commencing operation
krb5kdc: starting...
W20260619 11:14:04.825788 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:04 dist-test-slave-f1t5 krb5kdc[29650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:04.936797 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40681
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:05.040391 29663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:05.040680 29663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:05.040747 29663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:05.044627 29663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:05.044718 29663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:05.044768 29663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:05.044828 29663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:05.044867 29663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:05.049723 29663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:05.050727 29663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:05.051698 29663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:05.059491 29668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:05.059530 29669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:05.059530 29671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:05.059695 29663 server_base.cc:1034] running on GCE node
I20260619 11:14:05.060379 29663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:05.061400 29663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:05.062556 29663 hybrid_clock.cc:648] HybridClock initialized: now 1781867645062528 us; error 38 us; skew 500 ppm
Jun 19 11:14:05 dist-test-slave-f1t5 krb5kdc[29650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867645, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:05.065112 29663 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:05.065964 29663 webserver.cc:460] Webserver started at http://127.7.90.126:43001/ using document root <none> and password file <none>
I20260619 11:14:05.066499 29663 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:05.066588 29663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:05.066809 29663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:05.068725 29663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "bd9c4e35b930452ea04c7f10e1e5cba7"
format_stamp: "Formatted at 2026-06-19 11:14:05 on dist-test-slave-f1t5"
server_key: "d284f82319ea8c322288330c65a2d3e0"
server_key_iv: "14da28d27145cde0b33aaa2e7aa883b4"
server_key_version: "encryptionkey@0"
I20260619 11:14:05.069275 29663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "bd9c4e35b930452ea04c7f10e1e5cba7"
format_stamp: "Formatted at 2026-06-19 11:14:05 on dist-test-slave-f1t5"
server_key: "d284f82319ea8c322288330c65a2d3e0"
server_key_iv: "14da28d27145cde0b33aaa2e7aa883b4"
server_key_version: "encryptionkey@0"
I20260619 11:14:05.072714 29663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:14:05.075052 29678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:05.076246 29663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:14:05.076388 29663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "bd9c4e35b930452ea04c7f10e1e5cba7"
format_stamp: "Formatted at 2026-06-19 11:14:05 on dist-test-slave-f1t5"
server_key: "d284f82319ea8c322288330c65a2d3e0"
server_key_iv: "14da28d27145cde0b33aaa2e7aa883b4"
server_key_version: "encryptionkey@0"
I20260619 11:14:05.076493 29663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:05.080811 29663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:05.083688 29663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:05.083878 29663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:05.090299 29663 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:40681
I20260619 11:14:05.090355 29730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:40681 every 8 connection(s)
I20260619 11:14:05.091205 29663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:05.091933 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29663
I20260619 11:14:05.092067 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:05.092406 7529 external_mini_cluster.cc:1428] Setting key f8aed20933c0a61808a219264f88f9ca
I20260619 11:14:05.094401 29731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:05.097736 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:05.093797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33479 (local address 127.7.90.126:40681)
0619 11:14:05.094254 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:05.094271 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:05.094322 (+ 51us) server_negotiation.cc:407] Connection header received
0619 11:14:05.095109 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:05.095130 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:05.095461 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:05.095803 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:05.096420 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:05.096425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:05.096440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:05.096450 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:05.096502 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:05.096526 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:05.096656 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:05.096719 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:14:05.096860 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":128,"threads_started":1}
I20260619 11:14:05.100216 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:05.100476 29731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: Bootstrap starting.
I20260619 11:14:05.103070 29731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:05.103861 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:05.102330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51993 (local address 127.7.90.126:40681)
0619 11:14:05.102442 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:05.102446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:05.102672 (+ 226us) server_negotiation.cc:407] Connection header received
0619 11:14:05.102769 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:05.102771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:05.102797 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:05.102854 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:05.103521 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:05.103524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:05.103526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:14:05.103535 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:05.103557 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:14:05.103566 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:05.103631 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:05.103759 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 11:14:05.103784 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:14:05.104018 29731 log.cc:824] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:05.105938 29731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: No bootstrap required, opened a new log
I20260619 11:14:05.109211 29731 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } }
I20260619 11:14:05.109476 29731 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:14:05.109552 29731 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9c4e35b930452ea04c7f10e1e5cba7, State: Initialized, Role: FOLLOWER
I20260619 11:14:05.109997 29731 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } }
I20260619 11:14:05.110114 29731 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:14:05.110157 29731 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:14:05.110224 29731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:05.110956 29731 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } }
I20260619 11:14:05.111222 29731 leader_election.cc:302] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9c4e35b930452ea04c7f10e1e5cba7; no voters:
I20260619 11:14:05.111445 29731 leader_election.cc:288] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:14:05.111621 29741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:05.111914 29741 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [term 1 LEADER]: Becoming Leader. State: Replica: bd9c4e35b930452ea04c7f10e1e5cba7, State: Running, Role: LEADER
I20260619 11:14:05.112249 29741 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } }
I20260619 11:14:05.112632 29731 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:05.113935 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } } }
I20260619 11:14:05.114054 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:05.114362 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9c4e35b930452ea04c7f10e1e5cba7. Latest consensus state: current_term: 1 leader_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9c4e35b930452ea04c7f10e1e5cba7" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 40681 } } }
I20260619 11:14:05.114454 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7 [sys.catalog]: This master's current role is: LEADER
W20260619 11:14:05.117594 29756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:05.117677 29756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:05.117753 29751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:05.118417 29751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:05.124240 29751 catalog_manager.cc:1347] Generated new cluster ID: f4b4575aede6470eb59c8c97f3339343
I20260619 11:14:05.124300 29751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:05.133157 29751 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:05.133445 29751 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:05.144004 29751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9c4e35b930452ea04c7f10e1e5cba7: Generated new TSK 0
I20260619 11:14:05.144646 29751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:14:05.148837 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29663
2026-06-19T11:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1350 ms)
[----------] 18 tests from AuthTokenIssuingTest (24784 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:14:05 dist-test-slave-f1t5 krb5kdc[29762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:14:05 dist-test-slave-f1t5 krb5kdc[29762](info): set up 2 sockets
Jun 19 11:14:05 dist-test-slave-f1t5 krb5kdc[29762](info): commencing operation
krb5kdc: starting...
W20260619 11:14:06.169694 7529 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:06.284943 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38073
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:06.388365 29775 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:06.388654 29775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:06.388725 29775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:06.388808 29775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:06.392731 29775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:06.392822 29775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:06.392861 29775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:06.392931 29775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:06.392971 29775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:06.398010 29775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:06.399001 29775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:06.399998 29775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:06.407963 29783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:06.407958 29781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:06.407960 29780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:06.408483 29775 server_base.cc:1034] running on GCE node
I20260619 11:14:06.409015 29775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:06.410188 29775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:06.411370 29775 hybrid_clock.cc:648] HybridClock initialized: now 1781867646411340 us; error 42 us; skew 500 ppm
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:06.413964 29775 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:06.414719 29775 webserver.cc:460] Webserver started at http://127.7.90.126:41191/ using document root <none> and password file <none>
I20260619 11:14:06.415194 29775 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:06.415246 29775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:06.415409 29775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:06.417271 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "b6a73c8bfa4344c79afdf19ce4904b2e"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "c5e12614dbc672f327138d33334c2eb9"
server_key_iv: "bdca6c778b5efeaf62c15f776fac6446"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.417824 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "b6a73c8bfa4344c79afdf19ce4904b2e"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "c5e12614dbc672f327138d33334c2eb9"
server_key_iv: "bdca6c778b5efeaf62c15f776fac6446"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.421326 29775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 11:14:06.423615 29790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:06.424741 29775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:14:06.424837 29775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "b6a73c8bfa4344c79afdf19ce4904b2e"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "c5e12614dbc672f327138d33334c2eb9"
server_key_iv: "bdca6c778b5efeaf62c15f776fac6446"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.424914 29775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:06.452739 29775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:06.455519 29775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:06.455667 29775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:06.462401 29775 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:38073
I20260619 11:14:06.462692 29842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:38073 every 8 connection(s)
I20260619 11:14:06.463372 29775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:06.466096 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:06.469585 29843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.470261 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29775
I20260619 11:14:06.470399 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:06.470695 7529 external_mini_cluster.cc:1428] Setting key efcb0c3ef1ec58d90d39a71919660493
WARNING: no policy specified for kudu/127.7.90.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.125@KRBTEST.COM" created.
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:06.488529 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.471849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.7.90.126:38073)
0619 11:14:06.472226 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.472244 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.472595 (+ 351us) server_negotiation.cc:407] Connection header received
0619 11:14:06.473973 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.473977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.474144 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.474326 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.475126 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.475965 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.477011 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.477428 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.477614 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.482297 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.482303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.482315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.482345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.484392 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.484981 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.484989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.484993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.485294 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.485760 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.485766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.485769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.486303 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.486473 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.487119 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.487398 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260619 11:14:06.488530 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.471850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38073 (local address 127.0.0.1:55010)
0619 11:14:06.472411 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.472464 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.473646 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.474326 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.474331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.474756 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.474985 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.474997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.476091 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.476094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.476896 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.476905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.477590 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.477594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.477709 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.478291 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.478305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.482164 (+ 3859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.484480 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.484491 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.484504 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.484813 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.485442 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.485449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.485454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.485639 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.486487 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.486495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.486800 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.486989 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.487529 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":160,"spinlock_wait_cycles":10112,"thread_start_us":148,"threads_started":1}
I20260619 11:14:06.491436 29843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.492129 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.491904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54664)
0619 11:14:06.492014 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.492078 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 11:14:06.492321 29792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
I20260619 11:14:06.493167 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.492912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54680)
0619 11:14:06.493062 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.493112 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 11:14:06.493292 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:34743: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
I20260619 11:14:06.497897 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:34743
--webserver_interface=127.7.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:14:06.542570 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } attempt: 1
I20260619 11:14:06.543484 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.543147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54690)
0619 11:14:06.543316 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.543395 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 11:14:06.544217 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.543999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54694)
0619 11:14:06.544103 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.544154 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:14:06.544391 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:34743: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
W20260619 11:14:06.602867 29850 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:06.603096 29850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:06.603117 29850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:06.603235 29850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:06.607131 29850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:06.607175 29850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:06.607194 29850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:06.607210 29850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:06.607267 29850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:06.612238 29850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:06.612568 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } attempt: 2
I20260619 11:14:06.613310 29850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:06.613425 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.613105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54698)
0619 11:14:06.613250 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.613315 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 11:14:06.614097 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.613909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54702)
0619 11:14:06.614002 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.614045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 11:14:06.614255 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:34743: Network error: Client connection negotiation failed: client connection to 127.7.90.125:34743: connect: Connection refused (error 111)
I20260619 11:14:06.614506 29850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:06.622520 29855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:06.622567 29856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:06.622651 29850 server_base.cc:1034] running on GCE node
W20260619 11:14:06.622577 29858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:06.623378 29850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:06.624423 29850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:06.625620 29850 hybrid_clock.cc:648] HybridClock initialized: now 1781867646625595 us; error 29 us; skew 500 ppm
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:06.628487 29850 init.cc:377] Logged in from keytab as kudu/127.7.90.125@KRBTEST.COM (short username kudu)
I20260619 11:14:06.629387 29850 webserver.cc:460] Webserver started at http://127.7.90.125:44869/ using document root <none> and password file <none>
I20260619 11:14:06.629953 29850 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:06.630048 29850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:06.630256 29850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:06.632177 29850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data/instance:
uuid: "7b02a00c1db344af83c7cad3911f8aa7"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "3d15fe15acc6a82c6baecd25dd641b19"
server_key_iv: "d19404fd1b33ceb1aa080aa1ab6996f0"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.632754 29850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal/instance:
uuid: "7b02a00c1db344af83c7cad3911f8aa7"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "3d15fe15acc6a82c6baecd25dd641b19"
server_key_iv: "d19404fd1b33ceb1aa080aa1ab6996f0"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.636547 29850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 11:14:06.638926 29865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:06.640120 29850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:14:06.640265 29850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "7b02a00c1db344af83c7cad3911f8aa7"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "3d15fe15acc6a82c6baecd25dd641b19"
server_key_iv: "d19404fd1b33ceb1aa080aa1ab6996f0"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.640373 29850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:06.660825 29850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:06.664175 29850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:06.664409 29850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:06.671558 29850 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.125:34743
I20260619 11:14:06.671823 29917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.125:34743 every 8 connection(s)
I20260619 11:14:06.672526 29850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
I20260619 11:14:06.673273 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29850
I20260619 11:14:06.673405 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-1/wal/instance
I20260619 11:14:06.673785 7529 external_mini_cluster.cc:1428] Setting key 173fd43f86ec82064184e70ff74e3133
I20260619 11:14:06.675945 29918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:06.679653 29918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.124@KRBTEST.COM" created.
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:06.693315 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.681737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.7.90.126:38073)
0619 11:14:06.681900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.681904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.682683 (+ 779us) server_negotiation.cc:407] Connection header received
0619 11:14:06.683616 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.683619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.683684 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.683772 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.684816 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.685321 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.686548 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.686794 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.686914 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.689633 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.689636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.689638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.689663 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.691152 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.691748 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.691750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.691752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.691830 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.692208 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.692210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.692212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.692367 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.692434 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.693077 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.693192 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:14:06.694242 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.682090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38073 (local address 127.0.0.1:55022)
0619 11:14:06.682523 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.682571 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.683393 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.683929 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.683936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.684227 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.684671 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.684683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.685445 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.685453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.686426 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.686436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.686949 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.686954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.687114 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.687685 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.687701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.689436 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.691282 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.691290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.691307 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.691647 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.691925 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.691931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.691935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.692114 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.692466 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.692473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.692794 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.693027 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.693365 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260619 11:14:06.695712 29918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
I20260619 11:14:06.700300 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:34407
--webserver_interface=127.7.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:14:06.704303 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.696212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54706)
0619 11:14:06.696335 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.696351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.696437 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.696962 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.696965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.697129 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.697222 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.697227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.698277 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.698282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.698904 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.698912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.699507 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.699510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.699602 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.700032 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.700043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.701124 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.702890 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.702895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.702897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.703075 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.703356 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.703359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.703361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.703436 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.703836 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.703839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.703915 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.704027 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.704142 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 11:14:06.704361 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.696326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.7.90.125:34743)
0619 11:14:06.696595 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.696604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.696640 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:14:06.696683 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.696689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.696843 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.696971 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.697328 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.698160 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.699012 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.699399 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.699512 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.701249 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.701260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.701275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.701293 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.702769 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.703174 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.703179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.703181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.703254 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.703518 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.703523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.703524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.703732 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.703868 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.704101 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.704212 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260619 11:14:06.705413 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } attempt: 3
I20260619 11:14:06.707105 29918 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.707842 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.707535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45624)
0619 11:14:06.707672 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.707762 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 11:14:06.709000 29869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
I20260619 11:14:06.710021 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.709745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45634)
0619 11:14:06.709886 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.709945 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 11:14:06.710578 29918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:14:06.714221 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.706088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.7.90.125:34743)
0619 11:14:06.706204 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.706210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.706225 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:14:06.706262 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.706266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.706314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.706447 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.707050 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.707692 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.708887 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.709206 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.709304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.711509 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.711512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.711513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.711531 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.712731 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.713158 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.713160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.713161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.713222 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.713496 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.713498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.713499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.713720 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.713792 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.714057 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.714135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:14:06.714283 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.705888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54710)
0619 11:14:06.706015 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.706031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.706101 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.706604 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.706609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.706799 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.706937 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.706946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.707790 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.707796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.708769 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.708778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.709346 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.709351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.709514 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.710006 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.710026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.711403 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.712822 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.712825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.712827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.713050 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.713301 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.713304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.713306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.713408 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.713806 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.713809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.713911 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.714014 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.714132 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:14:06.715348 29843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.715952 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.715740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45648)
0619 11:14:06.715845 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.715887 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 11:14:06.716629 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.716410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45662)
0619 11:14:06.716547 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.716586 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 11:14:06.716794 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
I20260619 11:14:06.747947 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 1
I20260619 11:14:06.748633 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.748383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45666)
0619 11:14:06.748512 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.748570 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 11:14:06.749338 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.749140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45680)
0619 11:14:06.749239 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.749283 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:14:06.749500 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
I20260619 11:14:06.759930 29918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 1
I20260619 11:14:06.760632 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.760342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45692)
0619 11:14:06.760482 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.760544 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 11:14:06.761312 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.761066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45702)
0619 11:14:06.761197 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.761247 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 11:14:06.761449 29918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
W20260619 11:14:06.808983 29924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:06.809249 29924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:06.809311 29924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:06.809379 29924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:06.813019 29924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:06.813092 29924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:06.813138 29924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:06.813169 29924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:06.813197 29924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:06.818244 29924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:38073,127.7.90.125:34743,127.7.90.124:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:06.819141 29924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:06.820017 29924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:14:06.824679 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 2
I20260619 11:14:06.825485 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.825193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45706)
0619 11:14:06.825336 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.825400 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 11:14:06.826176 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.825943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45716)
0619 11:14:06.826065 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.826111 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:14:06.826319 29843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
W20260619 11:14:06.828078 29933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:06.828110 29930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:06.828200 29924 server_base.cc:1034] running on GCE node
W20260619 11:14:06.828361 29931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:06.828917 29924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:06.829646 29918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 2
I20260619 11:14:06.830101 29924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:06.830282 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.830031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45728)
0619 11:14:06.830145 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.830217 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 11:14:06.830964 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.830706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45744)
0619 11:14:06.830807 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.830908 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:14:06.831100 29918 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:34407: Network error: Client connection negotiation failed: client connection to 127.7.90.124:34407: connect: Connection refused (error 111)
I20260619 11:14:06.831264 29924 hybrid_clock.cc:648] HybridClock initialized: now 1781867646831232 us; error 42 us; skew 500 ppm
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:06.833770 29924 init.cc:377] Logged in from keytab as kudu/127.7.90.124@KRBTEST.COM (short username kudu)
I20260619 11:14:06.834561 29924 webserver.cc:460] Webserver started at http://127.7.90.124:43573/ using document root <none> and password file <none>
I20260619 11:14:06.835078 29924 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:06.835167 29924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:06.835379 29924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:06.837256 29924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data/instance:
uuid: "9a5ce353803848a2b88bb6bb8fda89bb"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "0c36665e80105acbcdae28980a022620"
server_key_iv: "479928b40bf71ae507d4bfe8b37dbe85"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.837814 29924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal/instance:
uuid: "9a5ce353803848a2b88bb6bb8fda89bb"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "0c36665e80105acbcdae28980a022620"
server_key_iv: "479928b40bf71ae507d4bfe8b37dbe85"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.841162 29924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:14:06.843361 29940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:06.844581 29924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:14:06.844705 29924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "9a5ce353803848a2b88bb6bb8fda89bb"
format_stamp: "Formatted at 2026-06-19 11:14:06 on dist-test-slave-f1t5"
server_key: "0c36665e80105acbcdae28980a022620"
server_key_iv: "479928b40bf71ae507d4bfe8b37dbe85"
server_key_version: "encryptionkey@0"
I20260619 11:14:06.844805 29924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:06.859263 29924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:06.861799 29924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:06.861975 29924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:06.868654 29924 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.124:34407
I20260619 11:14:06.868911 29992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.124:34407 every 8 connection(s)
I20260619 11:14:06.869540 29924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
I20260619 11:14:06.871991 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:06.875398 29993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.875634 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 29924
I20260619 11:14:06.875761 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867375844525-7529-0/minicluster-data/master-2/wal/instance
I20260619 11:14:06.876030 7529 external_mini_cluster.cc:1428] Setting key 261c4c74aa3a70e1e78402b220280c0a
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:14:06.889067 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.877147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.7.90.126:38073)
0619 11:14:06.877276 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.877279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.877960 (+ 681us) server_negotiation.cc:407] Connection header received
0619 11:14:06.878730 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.878733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.878777 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.878856 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.880130 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.880866 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.882198 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.882541 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.882641 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.885122 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.885125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.885128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.885159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.886843 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.887390 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.887396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.887401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.887490 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.887854 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.887856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.887859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.888091 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.888182 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.888632 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.888768 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:14:06.889694 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.877291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.7.90.124:34407)
0619 11:14:06.877778 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.877794 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.877830 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:14:06.878524 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.878544 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.878760 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.878925 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.879769 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.881051 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.882185 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.882750 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.882834 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.885630 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.885637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.885648 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.885672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.887308 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.887770 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.887774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.887775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.887869 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.888109 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.888114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.888118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.888395 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.888531 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.888805 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.888920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":43904,"thread_start_us":160,"threads_started":1}
I20260619 11:14:06.889700 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.877291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:38073 (local address 127.0.0.1:55032)
0619 11:14:06.877801 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.877844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.878524 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.879391 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.879399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.879832 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.879988 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.879997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.880963 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.880965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.882079 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.882088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.882672 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.882675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.882832 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.883401 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.883426 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.884983 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.886950 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.886957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.886964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.887288 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.887575 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.887578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.887580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.887759 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.888185 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.888189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.888393 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.888569 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.888857 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":42,"spinlock_wait_cycles":3584,"thread_start_us":146,"threads_started":1}
I20260619 11:14:06.890793 29993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:06.899798 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.891332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.7.90.125:34743)
0619 11:14:06.891470 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.891476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.891494 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:14:06.891537 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.891540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.891580 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.891668 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.892026 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.892541 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.893695 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.894021 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.894122 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.896137 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.896142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.896148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.896177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.898141 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.898691 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.898695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.898696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.898772 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.899008 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.899011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.899012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.899205 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.899327 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.899584 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.899670 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:14:06.899812 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.891219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:34743 (local address 127.0.0.1:54716)
0619 11:14:06.891374 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.891390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.891457 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.891677 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.891680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.891841 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.891915 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.891920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.892645 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.892651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.893564 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.893574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.894125 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.894128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.894195 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.894605 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.894619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.895966 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.898389 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.898391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.898393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.898586 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.898861 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.898863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.898864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.898927 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.899323 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.899325 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.899404 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.899521 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.899643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 11:14:06.900455 29993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:06.901227 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.891999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.7.90.126:38073)
0619 11:14:06.892090 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.892094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.892807 (+ 713us) server_negotiation.cc:407] Connection header received
0619 11:14:06.892909 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.892911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.892944 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.893004 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.893710 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.894395 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.895413 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.895772 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.895846 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.897986 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.897989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.897993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.898020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.899393 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.899808 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.899814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.899816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.899895 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.900172 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.900177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.900179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.900419 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.900624 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.901060 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.901142 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:14:06.909134 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.900907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45766)
0619 11:14:06.901430 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.901459 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.901596 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.901828 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.901831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.901981 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.902033 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.902037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.902966 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.902968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.903611 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.903619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.904086 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.904089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.904141 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.904544 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.904554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.905899 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.907629 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.907634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.907638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.907876 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.908140 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.908142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.908143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.908238 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.908687 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.908689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.908784 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.908902 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.909035 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448}
I20260619 11:14:06.909284 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.900922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.7.90.124:34407)
0619 11:14:06.901013 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.901017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.901510 (+ 493us) server_negotiation.cc:407] Connection header received
0619 11:14:06.901687 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.901691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.901732 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.901819 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.902133 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.902852 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.903712 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.903962 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.904074 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.906022 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.906024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.906026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.906046 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.907531 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.907970 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.907973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.907975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.908053 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.908330 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.908332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.908333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.908571 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.908691 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.908965 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.909105 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":27}
I20260619 11:14:06.911155 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.902359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.7.90.125:34743)
0619 11:14:06.902446 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.902450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.902463 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:14:06.902518 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.902520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.902550 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.902608 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.903241 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.903867 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.904795 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.905097 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.905173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.907989 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.907992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.907994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.908017 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.909680 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.910105 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.910109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.910112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.910209 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.910476 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.910477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.910478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.910610 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.910763 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.910936 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.911011 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:14:06.912482 29993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: Bootstrap starting.
I20260619 11:14:06.914330 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:06.914831 29993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:06.915525 29993 log.cc:824] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:06.915524 29817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55034
I20260619 11:14:06.917357 29993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: No bootstrap required, opened a new log
I20260619 11:14:06.920184 29993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.920418 29993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5ce353803848a2b88bb6bb8fda89bb, State: Initialized, Role: FOLLOWER
I20260619 11:14:06.920715 29993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.921141 29999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:06.921262 29999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:06.921962 29993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:06.922245 29918 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 3
I20260619 11:14:06.925453 29843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } attempt: 3
W20260619 11:14:06.927109 30010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:06.927223 30010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:06.927171 29924 master_runner.cc:186] Error getting master registration for 127.7.90.126:38073: OK, instance_id { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" instance_seqno: 1781867646460445 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:14:06.931072 29924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:14:06 dist-test-slave-f1t5 krb5kdc[29762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867646, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:14:06.936432 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.925949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45784)
0619 11:14:06.926574 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.926588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.926664 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.927401 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.927406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.927579 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.927684 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.927691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.928855 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.928859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.929793 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.929803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.930453 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.930468 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.930554 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.931083 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.931098 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.932581 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.934880 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.934884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.934886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.935078 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.935405 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.935409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.935489 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.935940 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.935944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.936035 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.936136 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.936261 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530}
I20260619 11:14:06.936460 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.926782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.7.90.124:34407)
0619 11:14:06.927115 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.927145 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.927158 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:14:06.927212 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.927215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.927275 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.927387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.927879 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.928728 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.929913 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.930345 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.930442 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.932740 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.932743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.932746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.932768 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.934779 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935176 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.935182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.935184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.935288 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935597 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.935602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.935603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.935804 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.935936 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.936214 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.936314 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260619 11:14:06.936708 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.922604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:34407 (local address 127.0.0.1:45768)
0619 11:14:06.922707 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:06.922720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:06.922785 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:06.923214 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:06.923217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:06.923400 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:06.923511 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.923520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.924567 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.924570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.925337 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:06.925347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.931222 (+ 5875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.931225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:06.931325 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.931750 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:06.931763 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:06.933093 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:06.935139 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.935145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.935384 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.935681 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:06.935686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:06.935766 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:06.936234 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:06.936237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:06.936327 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 11:14:06.936428 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 11:14:06.936557 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 11:14:06.937144 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.922665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.7.90.124:34407)
0619 11:14:06.922952 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.922956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.922974 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:14:06.923018 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.923022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.923062 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.923144 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.923668 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.924453 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.925447 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.925776 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.925870 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.933236 (+ 7366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.933239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.933242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.933265 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.935018 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935495 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.935499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.935500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.935585 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.935867 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.935871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.935872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.936099 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.936189 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.936905 (+ 716us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.937003 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260619 11:14:06.939864 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: Bootstrap starting.
I20260619 11:14:06.939873 29918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: Bootstrap starting.
I20260619 11:14:06.942296 29918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:06.942296 29843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:06.943086 29843 log.cc:824] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:06.943089 29918 log.cc:824] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:06.945029 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: No bootstrap required, opened a new log
I20260619 11:14:06.945359 29918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: No bootstrap required, opened a new log
I20260619 11:14:06.948019 29843 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.948247 29843 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a73c8bfa4344c79afdf19ce4904b2e, State: Initialized, Role: FOLLOWER
I20260619 11:14:06.948570 29843 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.949065 30015 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:06.949059 29918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.949249 29918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b02a00c1db344af83c7cad3911f8aa7, State: Initialized, Role: FOLLOWER
I20260619 11:14:06.949510 30015 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:06.949509 29918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.949887 29843 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:06.949921 29817 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:14:06.950002 29817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:06.950640 29918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:06.951761 29817 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.954367 29966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a5ce353803848a2b88bb6bb8fda89bb"
I20260619 11:14:06.954602 29966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:06.953802 29892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b02a00c1db344af83c7cad3911f8aa7"
W20260619 11:14:06.955852 30038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:06.955924 30038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:06.955927 29966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a73c8bfa4344c79afdf19ce4904b2e in term 1.
I20260619 11:14:06.956010 29892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [term 0 FOLLOWER]: Advancing to term 1
W20260619 11:14:06.957028 29796 tablet.cc:2369] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:14:06.957190 29892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a73c8bfa4344c79afdf19ce4904b2e in term 1.
I20260619 11:14:06.957619 29794 leader_election.cc:302] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5ce353803848a2b88bb6bb8fda89bb, b6a73c8bfa4344c79afdf19ce4904b2e; no voters:
I20260619 11:14:06.957880 30015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:06.958082 30015 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [term 1 LEADER]: Becoming Leader. State: Replica: b6a73c8bfa4344c79afdf19ce4904b2e, State: Running, Role: LEADER
I20260619 11:14:06.958417 30015 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } }
I20260619 11:14:06.958554 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:06.958655 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:06.952731 29817 leader_election.cc:288] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [CANDIDATE]: Term 1 election: Requested vote from peers 7b02a00c1db344af83c7cad3911f8aa7 (127.7.90.125:34743), 9a5ce353803848a2b88bb6bb8fda89bb (127.7.90.124:34407)
I20260619 11:14:06.960074 30015 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a73c8bfa4344c79afdf19ce4904b2e. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:06.960183 30015 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: This master's current role is: LEADER
W20260619 11:14:06.965657 30035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:06.965770 30035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:06.966102 30042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:06.966810 29850 master_runner.cc:263] No leader master found from master 7b02a00c1db344af83c7cad3911f8aa7
I20260619 11:14:06.966877 29850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:14:06.967821 30042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:06.980918 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.969552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.7.90.124:34407)
0619 11:14:06.969669 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.969674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.971691 (+ 2017us) server_negotiation.cc:407] Connection header received
0619 11:14:06.971897 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.971900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.971942 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.971990 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.973004 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.973823 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.975072 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.975398 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.975477 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.977300 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.977304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.977306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.977333 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.979037 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.979514 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.979517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.979519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.979597 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.979930 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.979934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.979936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.980135 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.980246 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.980628 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.980742 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:14:06.986884 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.970133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.7.90.126:38073)
0619 11:14:06.970249 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.970253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.973740 (+ 3487us) server_negotiation.cc:407] Connection header received
0619 11:14:06.973898 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.973901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.973943 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.973990 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.977616 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.978351 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.979432 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.979770 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.979837 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.981463 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.981467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.981470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.981495 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.983143 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.985614 (+ 2471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.985621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.985623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.985708 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.986002 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.986006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.986008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.986197 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.986299 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.986625 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.986732 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:14:06.990265 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:06.971350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.7.90.125:34743)
0619 11:14:06.971460 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:06.971464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:06.981675 (+ 10211us) server_negotiation.cc:407] Connection header received
0619 11:14:06.981854 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:06.981857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:06.981902 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:06.981950 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:06.982679 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.983441 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.984542 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:06.984878 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:06.984956 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:06.986506 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:06.986510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:06.986513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:06.986537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:06.988379 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.988919 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.988923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.988926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.989022 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:06.989306 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:06.989310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:06.989311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:06.989503 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:06.989637 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:06.990002 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 11:14:06.990113 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:14:06.995896 29892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [term 1 FOLLOWER]: Refusing update from remote peer b6a73c8bfa4344c79afdf19ce4904b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:14:06.996706 29966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [term 1 FOLLOWER]: Refusing update from remote peer b6a73c8bfa4344c79afdf19ce4904b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:14:06.997769 30015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:14:06.998389 30015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:14:07.007485 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a73c8bfa4344c79afdf19ce4904b2e. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.007602 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:07.008944 29999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a73c8bfa4344c79afdf19ce4904b2e. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.009063 29999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:07.018016 30015 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.018141 30015 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:07.018252 30049 mvcc.cc:205] Tried to move back new op lower bound from 7298529882086076416 to 7298529881945083904. Current Snapshot: MvccSnapshot[applied={T|T < 7298529882086076416}]
I20260619 11:14:07.019374 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.019482 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:07.021710 30040 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.021837 30040 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:07.023757 30042 catalog_manager.cc:1347] Generated new cluster ID: cc6d3bdc1f714a459f1b4bb492a0dfea
I20260619 11:14:07.023820 30042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:07.024750 29999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a73c8bfa4344c79afdf19ce4904b2e" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 38073 } } peers { permanent_uuid: "7b02a00c1db344af83c7cad3911f8aa7" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 34743 } } peers { permanent_uuid: "9a5ce353803848a2b88bb6bb8fda89bb" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 34407 } } }
I20260619 11:14:07.024864 29999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:07.045403 30042 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:07.046676 30042 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:07.059917 30042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6a73c8bfa4344c79afdf19ce4904b2e: Generated new TSK 0
I20260619 11:14:07.060770 30042 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:14:07.091102 29997 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.088114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.7.90.124:34407)
0619 11:14:07.088235 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.088241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.090070 (+ 1829us) server_negotiation.cc:407] Connection header received
0619 11:14:07.090252 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.090255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.090298 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.090346 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.090973 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45808: BlockingRecv error: recv got EOF from 127.0.0.1:45808 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:14:07.092804 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.089450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.7.90.125:34743)
0619 11:14:07.089571 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.089575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.091789 (+ 2214us) server_negotiation.cc:407] Connection header received
0619 11:14:07.092084 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.092087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.092138 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.092190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.092695 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54740: BlockingRecv error: recv got EOF from 127.0.0.1:54740 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:14:07.093832 29845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.088828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.7.90.126:38073)
0619 11:14:07.088944 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.088948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.091579 (+ 2631us) server_negotiation.cc:407] Connection header received
0619 11:14:07.093059 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.093062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.093118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.093174 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.093723 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55050: BlockingRecv error: recv got EOF from 127.0.0.1:55050 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 11:14:07.098310 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.096798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.7.90.125:34743)
0619 11:14:07.096913 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.096917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.097347 (+ 430us) server_negotiation.cc:407] Connection header received
0619 11:14:07.097596 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.097599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.097668 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.097724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.098225 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54744: BlockingRecv error: recv got EOF from 127.0.0.1:54744 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 11:14:07.103125 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.101721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.7.90.125:34743)
0619 11:14:07.101814 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.101818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.102317 (+ 499us) server_negotiation.cc:407] Connection header received
0619 11:14:07.102490 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.102493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.102537 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.102585 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.103047 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54760: BlockingRecv error: recv got EOF from 127.0.0.1:54760 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:14:07.108454 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.107080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.7.90.125:34743)
0619 11:14:07.107171 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.107175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.107623 (+ 448us) server_negotiation.cc:407] Connection header received
0619 11:14:07.107802 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.107805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.107860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.107924 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.108375 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54766: BlockingRecv error: recv got EOF from 127.0.0.1:54766 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:14:07.116463 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.115160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.7.90.125:34743)
0619 11:14:07.115296 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.115300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.115717 (+ 417us) server_negotiation.cc:407] Connection header received
0619 11:14:07.115870 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.115873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.115924 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.115968 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.116381 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54776: BlockingRecv error: recv got EOF from 127.0.0.1:54776 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 11:14:07.128167 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.126674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.7.90.125:34743)
0619 11:14:07.126769 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.126773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.127271 (+ 498us) server_negotiation.cc:407] Connection header received
0619 11:14:07.127503 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.127506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.127552 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.127597 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.128076 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54786: BlockingRecv error: recv got EOF from 127.0.0.1:54786 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 11:14:07.147238 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.146145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.7.90.125:34743)
0619 11:14:07.146245 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.146249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.146478 (+ 229us) server_negotiation.cc:407] Connection header received
0619 11:14:07.146637 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.146640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.146685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.146767 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.147144 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54796: BlockingRecv error: recv got EOF from 127.0.0.1:54796 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:14:07.182248 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.181124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.7.90.125:34743)
0619 11:14:07.181281 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.181285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.181465 (+ 180us) server_negotiation.cc:407] Connection header received
0619 11:14:07.181611 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.181614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.181660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.181740 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.182119 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54810: BlockingRecv error: recv got EOF from 127.0.0.1:54810 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 11:14:07.249529 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.248424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.7.90.125:34743)
0619 11:14:07.248570 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.248573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.248817 (+ 244us) server_negotiation.cc:407] Connection header received
0619 11:14:07.248878 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.248881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.248930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.249022 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.249420 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54812: BlockingRecv error: recv got EOF from 127.0.0.1:54812 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:14:07.380761 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.379711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.7.90.125:34743)
0619 11:14:07.379865 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.379871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.380032 (+ 161us) server_negotiation.cc:407] Connection header received
0619 11:14:07.380124 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.380127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.380178 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.380260 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.380652 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54826: BlockingRecv error: recv got EOF from 127.0.0.1:54826 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 11:14:07.639911 29923 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:07.638962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.7.90.125:34743)
0619 11:14:07.639100 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:07.639104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:07.639188 (+ 84us) server_negotiation.cc:407] Connection header received
0619 11:14:07.639305 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:07.639308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:07.639356 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:07.639435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:07.639820 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54840: BlockingRecv error: recv got EOF from 127.0.0.1:54840 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:14:07.928624 30010 catalog_manager.cc:1259] Loaded cluster ID: cc6d3bdc1f714a459f1b4bb492a0dfea
I20260619 11:14:07.928720 30010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: loading cluster ID for follower catalog manager: success
I20260619 11:14:07.930398 30010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: acquiring CA information for follower catalog manager: success
I20260619 11:14:07.931489 30010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a5ce353803848a2b88bb6bb8fda89bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:14:07.967085 30035 catalog_manager.cc:1259] Loaded cluster ID: cc6d3bdc1f714a459f1b4bb492a0dfea
I20260619 11:14:07.967162 30035 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: loading cluster ID for follower catalog manager: success
I20260619 11:14:07.968807 30035 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: acquiring CA information for follower catalog manager: success
I20260619 11:14:07.969861 30035 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b02a00c1db344af83c7cad3911f8aa7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:14:08.157713 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:08.154130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.7.90.125:34743)
0619 11:14:08.154366 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:08.154370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:08.154441 (+ 71us) server_negotiation.cc:407] Connection header received
0619 11:14:08.154561 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:08.154563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:08.154686 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:08.154792 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:08.155017 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.155538 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.156347 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.156565 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.156644 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:08.156921 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:08.157351 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:08.157440 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:08.157493 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 11:14:08.157593 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260619 11:14:08.164458 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:08.160987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.7.90.124:34407)
0619 11:14:08.161220 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:08.161223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:08.161271 (+ 48us) server_negotiation.cc:407] Connection header received
0619 11:14:08.161348 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:08.161350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:08.161456 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:08.161636 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:08.161800 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.162318 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.163097 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.163335 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.163435 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:08.163687 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:08.164152 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:08.164258 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:08.164287 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 11:14:08.164337 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260619 11:14:08.171309 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:08.167526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.7.90.125:34743)
0619 11:14:08.167740 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:08.167744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:08.167760 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:14:08.167877 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:08.167881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:08.167931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:08.168008 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:08.168268 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.168889 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.169801 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.170131 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.170231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:08.170491 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:08.170553 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:08.170628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:08.171087 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 11:14:08.171159 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 11:14:08.171439 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:08.167595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.7.90.124:34407)
0619 11:14:08.167858 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:08.167862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:08.168087 (+ 225us) server_negotiation.cc:407] Connection header received
0619 11:14:08.168258 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:08.168262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:08.168304 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:08.168383 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:08.168640 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.169373 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.170340 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.170648 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.170743 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:08.171006 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:08.171073 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:08.171145 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:08.171271 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 11:14:08.171319 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 11:14:08.172446 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:08.167398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.7.90.126:38073)
0619 11:14:08.168011 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:08.168016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:08.168029 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:14:08.168115 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:08.168118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:08.168287 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:08.168718 (+ 431us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:08.168842 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.169627 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.170750 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:08.171081 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:08.171204 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:08.171549 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:08.171894 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:08.171984 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:08.172196 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 11:14:08.172253 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":71,"threads_started":1}
I20260619 11:14:08.178639 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29775
I20260619 11:14:08.184827 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29850
I20260619 11:14:08.190235 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 29924
2026-06-19T11:14:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3043 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:14:08 dist-test-slave-f1t5 krb5kdc[30141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:14:08 dist-test-slave-f1t5 krb5kdc[30141](info): set up 2 sockets
Jun 19 11:14:08 dist-test-slave-f1t5 krb5kdc[30141](info): commencing operation
krb5kdc: starting...
W20260619 11:14:09.209852 7529 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/test-user.keytab.
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.90.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
Entry for principal kudu/127.7.90.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab.
I20260619 11:14:09.324389 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35525
--webserver_interface=127.7.90.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:14:09.428754 30154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:09.429078 30154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:09.429152 30154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:09.429229 30154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:09.433214 30154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:09.433300 30154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:09.433337 30154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:09.433399 30154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:09.433437 30154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:09.438915 30154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.126.keytab
--principal=kudu/127.7.90.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.126:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.90.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:09.439873 30154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:09.440812 30154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:14:09.448668 30160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:09.448695 30162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:09.448702 30159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:09.448927 30154 server_base.cc:1034] running on GCE node
I20260619 11:14:09.450421 30154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:09.451400 30154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:09.452581 30154 hybrid_clock.cc:648] HybridClock initialized: now 1781867649452567 us; error 27 us; skew 500 ppm
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:09.455262 30154 init.cc:377] Logged in from keytab as kudu/127.7.90.126@KRBTEST.COM (short username kudu)
I20260619 11:14:09.456107 30154 webserver.cc:460] Webserver started at http://127.7.90.126:40461/ using document root <none> and password file <none>
I20260619 11:14:09.456729 30154 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:09.456830 30154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:09.457119 30154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:09.459632 30154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data/instance:
uuid: "79bd15bc0951486f9c0888dd0ed07565"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "1f924fe1fcf4748ff3bb30578c44ea32"
server_key_iv: "1cf1c78c2140049ebff92ddf5f487972"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.460316 30154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance:
uuid: "79bd15bc0951486f9c0888dd0ed07565"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "1f924fe1fcf4748ff3bb30578c44ea32"
server_key_iv: "1cf1c78c2140049ebff92ddf5f487972"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.463973 30154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:14:09.466383 30169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:09.467535 30154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:14:09.467674 30154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal
uuid: "79bd15bc0951486f9c0888dd0ed07565"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "1f924fe1fcf4748ff3bb30578c44ea32"
server_key_iv: "1cf1c78c2140049ebff92ddf5f487972"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.467785 30154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:09.492419 30154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:09.495352 30154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:09.495553 30154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:09.502251 30154 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.126:35525
I20260619 11:14:09.502552 30221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.126:35525 every 8 connection(s)
I20260619 11:14:09.503173 30154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/data/info.pb
I20260619 11:14:09.505851 30222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:09.509481 30222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.510392 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 30154
I20260619 11:14:09.510526 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-0/wal/instance
I20260619 11:14:09.510828 7529 external_mini_cluster.cc:1428] Setting key 35b865cbd6de5ea5d9911a7da66ec018
WARNING: no policy specified for kudu/127.7.90.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.125@KRBTEST.COM" created.
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:09.529247 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.511951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.7.90.126:35525)
0619 11:14:09.512648 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.512694 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.512931 (+ 237us) server_negotiation.cc:407] Connection header received
0619 11:14:09.514499 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.514504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.514683 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.514854 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.515949 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.516822 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.518748 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.519091 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.519199 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.524023 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.524028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.524036 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.524056 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.525937 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.526474 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.526482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.526487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.526626 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.526991 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.526996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.526998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.527518 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.527665 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.528062 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.528484 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":41856,"thread_start_us":134,"threads_started":1}
I20260619 11:14:09.529256 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.511951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35525 (local address 127.0.0.1:59042)
0619 11:14:09.512645 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.512726 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.514201 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.514910 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.514919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.515428 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.515799 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.515818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.516994 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.517007 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.518633 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.518645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.519243 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.519249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.519415 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.519996 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.520022 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.523827 (+ 3805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.526059 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.526067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.526074 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.526347 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.526727 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.526730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.526731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.526847 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.527650 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.527655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.527787 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.527927 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.528384 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":66,"thread_start_us":132,"threads_started":1}
I20260619 11:14:09.531708 30222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.532429 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.532155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57316)
0619 11:14:09.532256 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.532348 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:14:09.532624 30170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
I20260619 11:14:09.533555 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.533251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57322)
0619 11:14:09.533406 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.533461 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 11:14:09.533695 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:44263: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
Entry for principal kudu/127.7.90.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab.
I20260619 11:14:09.538158 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:44263
--webserver_interface=127.7.90.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:14:09.582935 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } attempt: 1
I20260619 11:14:09.583806 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.583499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57330)
0619 11:14:09.583650 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.583718 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:14:09.584601 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.584335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57342)
0619 11:14:09.584474 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.584535 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 11:14:09.584740 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:44263: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
W20260619 11:14:09.642764 30229 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:09.642995 30229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:09.643018 30229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:09.643054 30229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:09.646857 30229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:09.646898 30229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:09.646935 30229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:09.646951 30229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:09.646966 30229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:09.651764 30229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.125.keytab
--principal=kudu/127.7.90.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.125:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.90.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:09.652591 30229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:09.652909 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } attempt: 2
I20260619 11:14:09.653689 30229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:14:09.653764 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.653399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57350)
0619 11:14:09.653608 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.653673 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 11:14:09.654426 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.654209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57356)
0619 11:14:09.654310 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.654368 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:14:09.654631 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.125:44263: Network error: Client connection negotiation failed: client connection to 127.7.90.125:44263: connect: Connection refused (error 111)
W20260619 11:14:09.661934 30234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:09.662045 30229 server_base.cc:1034] running on GCE node
W20260619 11:14:09.661948 30237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:09.661934 30235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:09.662786 30229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:09.663757 30229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:09.664945 30229 hybrid_clock.cc:648] HybridClock initialized: now 1781867649664931 us; error 33 us; skew 500 ppm
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:09.667658 30229 init.cc:377] Logged in from keytab as kudu/127.7.90.125@KRBTEST.COM (short username kudu)
I20260619 11:14:09.668507 30229 webserver.cc:460] Webserver started at http://127.7.90.125:33325/ using document root <none> and password file <none>
I20260619 11:14:09.669037 30229 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:09.669126 30229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:09.669351 30229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:09.671415 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data/instance:
uuid: "7b0ec54ba91747538e668bcfb258c5c9"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "85ebfbb7f0475fd35b590a16847accbc"
server_key_iv: "3c8f15777647a24b718b7912fddfb2fa"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.671959 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal/instance:
uuid: "7b0ec54ba91747538e668bcfb258c5c9"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "85ebfbb7f0475fd35b590a16847accbc"
server_key_iv: "3c8f15777647a24b718b7912fddfb2fa"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.675546 30229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 11:14:09.677868 30244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:09.679023 30229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:14:09.679155 30229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal
uuid: "7b0ec54ba91747538e668bcfb258c5c9"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "85ebfbb7f0475fd35b590a16847accbc"
server_key_iv: "3c8f15777647a24b718b7912fddfb2fa"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.679260 30229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:09.696075 30229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:09.699107 30229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:09.699294 30229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:09.706116 30229 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.125:44263
I20260619 11:14:09.706419 30296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.125:44263 every 8 connection(s)
I20260619 11:14:09.707072 30229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/data/info.pb
I20260619 11:14:09.709677 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:09.713501 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 30229
I20260619 11:14:09.713148 30297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.713654 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-1/wal/instance
I20260619 11:14:09.714059 7529 external_mini_cluster.cc:1428] Setting key afc1d19dda6d75f97173203cae50e696
WARNING: no policy specified for kudu/127.7.90.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.90.124@KRBTEST.COM" created.
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:09.728263 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.715361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.7.90.126:35525)
0619 11:14:09.715533 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.715541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.716392 (+ 851us) server_negotiation.cc:407] Connection header received
0619 11:14:09.717284 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.717288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.717345 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.717445 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.718632 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.719152 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.720256 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.720489 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.720602 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.724782 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.724785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.724787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.724810 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.726257 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.726769 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.726774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.726776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.726861 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.727193 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.727197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.727199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.727379 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.727479 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.727968 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.728088 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:14:09.729152 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.715768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35525 (local address 127.0.0.1:59044)
0619 11:14:09.716249 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.716291 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.717082 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.717625 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.717631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.718053 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.718494 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.718506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.719277 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.719282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.720124 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.720132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.720645 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.720649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.720816 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.721445 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.721462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.724598 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.726396 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.726401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.726416 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.726666 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.726952 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.726957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.726965 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.727099 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.727478 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.727485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.727725 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.727904 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.728210 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260619 11:14:09.730531 30297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
Entry for principal kudu/127.7.90.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab.
I20260619 11:14:09.738750 7529 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7DZYiV/build/debug/bin/kudu
/tmp/dist-test-task7DZYiV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:37895
--webserver_interface=127.7.90.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.90.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:14:09.739719 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.730988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57358)
0619 11:14:09.731094 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.731109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.731170 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.731744 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.731746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.731879 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.731965 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.731970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.732883 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.732886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.733415 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.733422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.734065 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.734067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.734198 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.734699 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.734719 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.736070 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.738027 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.738033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.738038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.738311 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.738620 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.738622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.738624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.738704 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.739219 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.739222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.739313 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.739429 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.739548 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:14:09.739779 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.731119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.7.90.125:44263)
0619 11:14:09.731381 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.731390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.731408 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:14:09.731442 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.731446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.731646 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.731754 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.732059 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.732778 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.733614 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.733969 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.734052 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.736198 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.736205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.736215 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.736235 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.737927 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.738433 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.738440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.738442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.738523 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.738802 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.738807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.738809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.739115 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.739256 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.739507 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.739626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260619 11:14:09.741662 30297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.742496 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.742166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43748)
0619 11:14:09.742300 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.742407 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 11:14:09.742703 30245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
I20260619 11:14:09.743562 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.743291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43758)
0619 11:14:09.743419 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.743484 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:14:09.743710 30297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
I20260619 11:14:09.745795 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } attempt: 3
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
I20260619 11:14:09.754632 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.746246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57368)
0619 11:14:09.746382 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.746397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.746520 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.746788 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.746791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.746963 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.747064 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.747069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.747816 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.747819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.748747 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.748757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.749217 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.749220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.749364 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.749901 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.749916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.751401 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.753123 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.753129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.753131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.753327 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.753679 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.753682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.753684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.753771 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.754129 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.754132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.754235 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.754350 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.754483 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:14:09.754631 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.746296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.7.90.125:44263)
0619 11:14:09.746389 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.746394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.746467 (+ 73us) server_negotiation.cc:407] Connection header received
0619 11:14:09.746610 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.746613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.746652 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.746771 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.747172 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.747678 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.748864 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.749111 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.749215 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.751526 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.751530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.751533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.751555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.753001 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.753422 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.753426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.753428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.753576 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.753870 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.753873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.753875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.754035 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.754138 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.754398 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.754496 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:14:09.755705 30222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.756319 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.756117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43772)
0619 11:14:09.756217 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.756270 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:14:09.756964 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.756716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43784)
0619 11:14:09.756848 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.756897 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 11:14:09.757102 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
I20260619 11:14:09.788264 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 1
I20260619 11:14:09.789248 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.788910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43798)
0619 11:14:09.789096 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.789165 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 11:14:09.790010 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.789794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43802)
0619 11:14:09.789903 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.789953 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:14:09.790175 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
I20260619 11:14:09.792959 30297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 1
I20260619 11:14:09.793651 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.793333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43804)
0619 11:14:09.793461 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.793579 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 11:14:09.794292 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.794094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43808)
0619 11:14:09.794194 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.794241 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:14:09.794418 30297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
W20260619 11:14:09.850345 30303 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:14:09.850704 30303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:14:09.850752 30303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:14:09.850811 30303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:14:09.856603 30303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:14:09.856652 30303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:14:09.856670 30303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:14:09.856688 30303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:14:09.856737 30303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:14:09.861841 30303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.90.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.90.126:35525,127.7.90.125:44263,127.7.90.124:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/krb5kdc/kudu_127.7.90.124.keytab
--principal=kudu/127.7.90.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.90.124:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.90.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:14:09.862567 30297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 2
I20260619 11:14:09.862936 30303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:14:09.863409 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.863100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43816)
0619 11:14:09.863253 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.863317 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 11:14:09.864145 30303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:14:09.864233 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.863995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43820)
0619 11:14:09.864109 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.864170 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 11:14:09.864343 30297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
I20260619 11:14:09.865358 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 2
I20260619 11:14:09.866024 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.865793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43834)
0619 11:14:09.865916 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.865965 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 11:14:09.866679 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.866472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43842)
0619 11:14:09.866582 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.866632 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:14:09.866829 30222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.90.124:37895: Network error: Client connection negotiation failed: client connection to 127.7.90.124:37895: connect: Connection refused (error 111)
W20260619 11:14:09.872593 30310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:14:09.872628 30312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:09.872659 30303 server_base.cc:1034] running on GCE node
W20260619 11:14:09.872880 30309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:14:09.873504 30303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:14:09.874660 30303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:14:09.875814 30303 hybrid_clock.cc:648] HybridClock initialized: now 1781867649875799 us; error 26 us; skew 500 ppm
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:14:09.878432 30303 init.cc:377] Logged in from keytab as kudu/127.7.90.124@KRBTEST.COM (short username kudu)
I20260619 11:14:09.879194 30303 webserver.cc:460] Webserver started at http://127.7.90.124:44297/ using document root <none> and password file <none>
I20260619 11:14:09.879702 30303 fs_manager.cc:359] Metadata directory not provided
I20260619 11:14:09.879786 30303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:14:09.880019 30303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:14:09.881983 30303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data/instance:
uuid: "523866ee61a5404cb3cbae78c45a77ad"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "76e9c16ca0a60d7387cad23a2cfe64ae"
server_key_iv: "99edf58bac740df12a05f2327909cd56"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.882524 30303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal/instance:
uuid: "523866ee61a5404cb3cbae78c45a77ad"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "76e9c16ca0a60d7387cad23a2cfe64ae"
server_key_iv: "99edf58bac740df12a05f2327909cd56"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.886169 30303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:14:09.888322 30319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:14:09.889397 30303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:14:09.889571 30303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data,/tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal
uuid: "523866ee61a5404cb3cbae78c45a77ad"
format_stamp: "Formatted at 2026-06-19 11:14:09 on dist-test-slave-f1t5"
server_key: "76e9c16ca0a60d7387cad23a2cfe64ae"
server_key_iv: "99edf58bac740df12a05f2327909cd56"
server_key_version: "encryptionkey@0"
I20260619 11:14:09.889698 30303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:14:09.915108 30303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:14:09.918414 30303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:14:09.918608 30303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:14:09.925172 30303 rpc_server.cc:307] RPC server started. Bound to: 127.7.90.124:37895
I20260619 11:14:09.925436 30371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.90.124:37895 every 8 connection(s)
I20260619 11:14:09.926136 30303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/data/info.pb
I20260619 11:14:09.928725 30372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:14:09.932354 30372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:14:09.934937 7529 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7DZYiV/build/debug/bin/kudu as pid 30303
I20260619 11:14:09.935076 7529 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7DZYiV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867375844525-7529-0/minicluster-data/master-2/wal/instance
I20260619 11:14:09.935393 7529 external_mini_cluster.cc:1428] Setting key 5cc3eb468a8c2759ade0f81006d44e84
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:14:09.945791 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.934085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.7.90.126:35525)
0619 11:14:09.934216 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.934220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.935035 (+ 815us) server_negotiation.cc:407] Connection header received
0619 11:14:09.936155 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.936160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.936223 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.936308 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.937422 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.937985 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.939083 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.939305 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.939412 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.941955 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.941964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.941966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.941987 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.943667 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.944192 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.944196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.944197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.944281 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.944684 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.944687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.944689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.944925 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.945018 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.945477 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.945639 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:14:09.946507 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.934407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.126:35525 (local address 127.0.0.1:59052)
0619 11:14:09.934857 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.934915 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.935887 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.936530 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.936537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.936856 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.937268 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.937292 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.938086 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.938093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.938965 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.938974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.939440 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.939444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.939665 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.940208 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.940227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.941811 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.943784 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.943791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.943797 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.944100 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.944395 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.944401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.944405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.944576 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.945024 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.945029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.945239 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.945412 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.945720 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":131,"thread_start_us":102,"threads_started":1}
I20260619 11:14:09.946511 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.936007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.7.90.124:37895)
0619 11:14:09.936345 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.936355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.936504 (+ 149us) server_negotiation.cc:407] Connection header received
0619 11:14:09.936660 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.936664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.936809 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.936967 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.937732 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.938563 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.939370 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.939774 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.939883 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.942444 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.942450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.942473 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.942499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.944213 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.944695 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.944700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.944701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.944772 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.945036 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.945039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.945041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.945271 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.945457 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.945662 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.945778 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":248,"spinlock_wait_cycles":12032,"thread_start_us":91,"threads_started":1}
I20260619 11:14:09.947882 30372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.126@KRBTEST.COM
I20260619 11:14:09.955503 30297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 3
I20260619 11:14:09.956921 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.948474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.7.90.125:44263)
0619 11:14:09.948594 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.948599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.948663 (+ 64us) server_negotiation.cc:407] Connection header received
0619 11:14:09.948923 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.948929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.948979 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.949060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.949497 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.950235 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.951119 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.951331 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.951415 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.953696 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.953701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.953708 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.953728 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.955192 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.955640 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.955643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.955645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.955725 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.956031 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.956038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.956040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.956243 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.956325 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.956662 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.956773 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:14:09.957136 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.948810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.7.90.126:35525)
0619 11:14:09.948916 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.948920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.948931 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:14:09.949014 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.949020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.949062 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.949129 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.949792 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.950475 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.951329 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.951583 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.951694 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.953761 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.953767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.953773 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.953794 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.955316 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.955757 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.955761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.955765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.956056 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.956349 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.956355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.956359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.956539 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.956745 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.956886 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.956994 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:14:09.957394 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.948333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.125:44263 (local address 127.0.0.1:57380)
0619 11:14:09.948592 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.948605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.948698 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.949070 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.949073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.949256 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.949379 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.949386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.950347 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.950349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.951018 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.951027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.951424 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.951426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.951489 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.951891 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.951909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.953567 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.955304 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.955309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.955312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.955528 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.955841 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.955844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.955845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.955921 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.956398 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.956403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.956482 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.957101 (+ 619us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.957224 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260619 11:14:09.958343 30372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.125@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.90.125@KRBTEST.COM
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.124@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:14:09.965131 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.955909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.7.90.124:37895)
0619 11:14:09.956022 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.956025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.956082 (+ 57us) server_negotiation.cc:407] Connection header received
0619 11:14:09.956185 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.956189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.956230 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.956308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.956684 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.957430 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.958711 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.959071 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.959169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.961261 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.961264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.961266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.961299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.963483 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.963907 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.963911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.963912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.963997 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.964318 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.964324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.964327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.964533 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.964660 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.964917 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.965009 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:14:09.965131 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.955888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43856)
0619 11:14:09.955998 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.956013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.956115 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.956320 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.956323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.956485 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.956549 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.956553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.957601 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.957605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.958348 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.958361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.959190 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.959194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.959300 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.959763 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.959777 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.961133 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.963591 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.963594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.963596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.963819 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.964094 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.964100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.964105 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.964194 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.964655 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.964658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.964751 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.964856 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.964975 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 11:14:09.966022 30222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } attempt: 3
I20260619 11:14:09.967684 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.958630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.7.90.125:44263)
0619 11:14:09.958721 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.958725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.958853 (+ 128us) server_negotiation.cc:407] Connection header received
0619 11:14:09.959015 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.959018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.959058 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.959107 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.959890 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.960337 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.961110 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.961321 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.961431 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.963857 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.963862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.963868 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.963893 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.965740 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.966296 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.966302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.966304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.966384 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.966681 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.966684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.966685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.966904 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.967116 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.967419 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.967523 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:14:09.968087 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.958766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43860)
0619 11:14:09.958841 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.958853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.958916 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.959348 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.959351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.959513 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.959583 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.959588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.960633 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.960637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.961361 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.961370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.962004 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.962008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.962098 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.962591 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.962608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.964209 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.966685 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.966688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.966690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.966899 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.967193 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.967196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.967198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.967284 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.967719 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.967723 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.967821 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.967913 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.967996 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 11:14:09.968592 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.7.90.124:37895)
0619 11:14:09.959153 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.959164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.959174 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:14:09.959201 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.959204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.959239 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.959688 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.959757 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.960517 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.961485 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.961876 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.962319 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.964336 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.964340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.964342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.964362 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.966588 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.967007 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.967010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.967011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.967089 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.967392 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.967395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.967396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.967594 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.967718 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.968333 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.968449 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
I20260619 11:14:09.969018 30297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: Bootstrap starting.
I20260619 11:14:09.972077 7529 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:14:09.972290 30297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:09.972766 30372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: Bootstrap starting.
I20260619 11:14:09.973182 30297 log.cc:824] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:09.973263 30196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79bd15bc0951486f9c0888dd0ed07565"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59066
Jun 19 11:14:09 dist-test-slave-f1t5 krb5kdc[30141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867649, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.90.126@KRBTEST.COM for kudu/127.7.90.124@KRBTEST.COM
I20260619 11:14:09.976168 30297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: No bootstrap required, opened a new log
I20260619 11:14:09.976342 30372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:09.977164 30372 log.cc:824] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:09.978439 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.969816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.7.90.124:37895)
0619 11:14:09.969955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:09.969959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:09.969972 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:14:09.970002 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:09.970005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:09.970045 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:09.970122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:09.970603 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.971318 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.972396 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.972709 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.973168 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.974993 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:09.974999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:09.975003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:09.975026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:09.976688 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.977201 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.977204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.977206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.977284 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:09.977631 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:09.977636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:09.977637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:09.977847 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:09.977956 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:09.978208 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 11:14:09.978312 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 11:14:09.978439 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:09.966366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.90.124:37895 (local address 127.0.0.1:43864)
0619 11:14:09.966585 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 11:14:09.966600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:14:09.966668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:14:09.970160 (+ 3492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:14:09.970163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:14:09.970349 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:14:09.970479 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.970488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.971446 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.971450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.972212 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:14:09.972223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:14:09.972855 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:14:09.972858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:14:09.972935 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:09.973471 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:14:09.973482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:14:09.974843 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:14:09.976823 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.976827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.976828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.977083 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.977390 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:14:09.977393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:14:09.977395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:14:09.977487 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:14:09.977948 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:14:09.977951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:14:09.978044 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 11:14:09.978148 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:14:09.978287 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 11:14:09.980005 30297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.980206 30297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0ec54ba91747538e668bcfb258c5c9, State: Initialized, Role: FOLLOWER
I20260619 11:14:09.980496 30297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.980938 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:09.981124 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:09.981304 30222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: Bootstrap starting.
I20260619 11:14:09.981851 30297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:09.983762 30222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: Neither blocks nor log segments found. Creating new log.
I20260619 11:14:09.984552 30222 log.cc:824] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: Log is configured to *not* fsync() on all Append() calls
I20260619 11:14:09.985672 30372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: No bootstrap required, opened a new log
W20260619 11:14:09.987334 30391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:09.987651 30391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:09.985363 30229 master_runner.cc:186] Error getting master registration for 127.7.90.126:35525: OK, instance_id { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" instance_seqno: 1781867649500430 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:14:09.988915 30229 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:14:09.989413 30222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: No bootstrap required, opened a new log
I20260619 11:14:09.989655 30372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.989887 30372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523866ee61a5404cb3cbae78c45a77ad, State: Initialized, Role: FOLLOWER
I20260619 11:14:09.990202 30372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.990748 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:09.991031 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:09.991410 30372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:09.992393 30222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.992614 30222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bd15bc0951486f9c0888dd0ed07565, State: Initialized, Role: FOLLOWER
I20260619 11:14:09.992915 30222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:09.994035 30403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:09.994165 30403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:09.994206 30222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:14:09.994349 30196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:14:09.994467 30196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:09.995671 30196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
W20260619 11:14:09.998842 30417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:09.998970 30417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 11:14:09.999135 30406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:14:09.999203 30406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:14:09.999418 30271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79bd15bc0951486f9c0888dd0ed07565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b0ec54ba91747538e668bcfb258c5c9"
I20260619 11:14:09.999639 30271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:10.000648 30271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bd15bc0951486f9c0888dd0ed07565 in term 1.
I20260619 11:14:10.001031 30170 leader_election.cc:302] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79bd15bc0951486f9c0888dd0ed07565, 7b0ec54ba91747538e668bcfb258c5c9; no voters:
I20260619 11:14:10.001047 30346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79bd15bc0951486f9c0888dd0ed07565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "523866ee61a5404cb3cbae78c45a77ad"
I20260619 11:14:10.001245 30346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:14:10.001307 30403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:14:10.001540 30403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [term 1 LEADER]: Becoming Leader. State: Replica: 79bd15bc0951486f9c0888dd0ed07565, State: Running, Role: LEADER
I20260619 11:14:10.001837 30403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } }
I20260619 11:14:10.002630 30346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bd15bc0951486f9c0888dd0ed07565 in term 1.
I20260619 11:14:09.996543 30196 leader_election.cc:288] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [CANDIDATE]: Term 1 election: Requested vote from peers 7b0ec54ba91747538e668bcfb258c5c9 (127.7.90.125:44263), 523866ee61a5404cb3cbae78c45a77ad (127.7.90.124:37895)
W20260619 11:14:09.997043 30175 tablet.cc:2369] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:14:10.005404 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79bd15bc0951486f9c0888dd0ed07565. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.005635 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:09.994786 30303 master_runner.cc:186] Error getting master registration for 127.7.90.126:35525: OK, instance_id { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" instance_seqno: 1781867649500430 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:14:10.005961 30303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:14:10.007308 30154 master_runner.cc:263] No leader master found from master 79bd15bc0951486f9c0888dd0ed07565
I20260619 11:14:10.007364 30154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:14:10.014528 30426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:14:10.015528 30426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:14:10.017403 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:10.006304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.7.90.125:44263)
0619 11:14:10.006420 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.006426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.007043 (+ 617us) server_negotiation.cc:407] Connection header received
0619 11:14:10.007227 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.007231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.007279 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.007330 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.008325 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.009144 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.010396 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.010728 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.010803 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:10.012649 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:10.012653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:10.012656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:10.012682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:10.014660 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.016015 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.016019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.016021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.016106 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.016417 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.016420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.016422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.016637 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:10.016736 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:10.017120 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 11:14:10.017243 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:14:10.022166 30346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [term 1 FOLLOWER]: Refusing update from remote peer 79bd15bc0951486f9c0888dd0ed07565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:14:10.022241 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:10.006284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.7.90.126:35525)
0619 11:14:10.006413 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.006419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.010071 (+ 3652us) server_negotiation.cc:407] Connection header received
0619 11:14:10.010239 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.010243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.010290 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.010348 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.011358 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.012114 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.013248 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.013654 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.013731 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:10.018222 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:10.018226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:10.018229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:10.018256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:10.019995 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.020852 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.020855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.020857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.020955 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.021263 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.021267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.021268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.021485 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:10.021611 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:10.021974 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 11:14:10.022092 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:14:10.023267 30420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:14:10.024490 30271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [term 1 FOLLOWER]: Refusing update from remote peer 79bd15bc0951486f9c0888dd0ed07565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:14:10.025427 30420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:14:10.028928 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79bd15bc0951486f9c0888dd0ed07565. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.029063 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:10.029718 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79bd15bc0951486f9c0888dd0ed07565. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.029876 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:10.032660 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.032765 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:10.033283 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:10.006960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.7.90.124:37895)
0619 11:14:10.007062 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.007065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.020620 (+ 13555us) server_negotiation.cc:407] Connection header received
0619 11:14:10.020729 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.020732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.020785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.020871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.025610 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.026350 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.027416 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:10.027730 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:10.027804 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:10.029624 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:14:10.029630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:14:10.029634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:14:10.029663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:14:10.031379 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.031877 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.031882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.031883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.031964 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:14:10.032300 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:14:10.032303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:14:10.032305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:14:10.032520 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:14:10.032627 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:10.032973 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 11:14:10.033093 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:14:10.034065 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.034157 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:10.034727 30433 mvcc.cc:205] Tried to move back new op lower bound from 7298529894482436096 to 7298529894410801152. Current Snapshot: MvccSnapshot[applied={T|T < 7298529894482436096}]
I20260619 11:14:10.034905 30430 mvcc.cc:205] Tried to move back new op lower bound from 7298529894482436096 to 7298529894410801152. Current Snapshot: MvccSnapshot[applied={T|T < 7298529894482436096}]
I20260619 11:14:10.038327 30426 catalog_manager.cc:1347] Generated new cluster ID: ff3f90b0e2d14bb689d1b08dd6c23106
I20260619 11:14:10.038385 30426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:14:10.046152 30403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.046270 30403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565 [sys.catalog]: This master's current role is: LEADER
I20260619 11:14:10.047645 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79bd15bc0951486f9c0888dd0ed07565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bd15bc0951486f9c0888dd0ed07565" member_type: VOTER last_known_addr { host: "127.7.90.126" port: 35525 } } peers { permanent_uuid: "7b0ec54ba91747538e668bcfb258c5c9" member_type: VOTER last_known_addr { host: "127.7.90.125" port: 44263 } } peers { permanent_uuid: "523866ee61a5404cb3cbae78c45a77ad" member_type: VOTER last_known_addr { host: "127.7.90.124" port: 37895 } } }
I20260619 11:14:10.047737 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:14:10.048269 30431 mvcc.cc:205] Tried to move back new op lower bound from 7298529894482436096 to 7298529894410801152. Current Snapshot: MvccSnapshot[applied={T|T < 7298529894482436096}]
I20260619 11:14:10.058604 30426 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:14:10.059939 30426 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:14:10.075937 30426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79bd15bc0951486f9c0888dd0ed07565: Generated new TSK 0
I20260619 11:14:10.076843 30426 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:14:10.109701 30226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.107796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.7.90.126:35525)
0619 11:14:10.107925 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.107930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.108587 (+ 657us) server_negotiation.cc:407] Connection header received
0619 11:14:10.108828 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.108832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.108884 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.108937 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.109563 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59070: BlockingRecv error: recv got EOF from 127.0.0.1:59070 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:14:10.112146 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.110492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.7.90.125:44263)
0619 11:14:10.110599 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.110621 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.111217 (+ 596us) server_negotiation.cc:407] Connection header received
0619 11:14:10.111382 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.111385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.111432 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.111482 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.111986 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57392: BlockingRecv error: recv got EOF from 127.0.0.1:57392 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:14:10.113724 30377 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.111021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.7.90.124:37895)
0619 11:14:10.111128 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.111132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.112853 (+ 1721us) server_negotiation.cc:407] Connection header received
0619 11:14:10.113017 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.113020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.113068 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.113118 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.113610 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43882: BlockingRecv error: recv got EOF from 127.0.0.1:43882 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 11:14:10.117834 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.116391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.7.90.125:44263)
0619 11:14:10.116486 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.116490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.116876 (+ 386us) server_negotiation.cc:407] Connection header received
0619 11:14:10.117088 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.117091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.117164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.117212 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.117731 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57406: BlockingRecv error: recv got EOF from 127.0.0.1:57406 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:14:10.122654 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.121335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.7.90.125:44263)
0619 11:14:10.121426 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.121430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.121881 (+ 451us) server_negotiation.cc:407] Connection header received
0619 11:14:10.122051 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.122053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.122095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.122140 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.122568 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57410: BlockingRecv error: recv got EOF from 127.0.0.1:57410 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:14:10.128119 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.126787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.7.90.125:44263)
0619 11:14:10.126879 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.126884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.127302 (+ 418us) server_negotiation.cc:407] Connection header received
0619 11:14:10.127467 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.127470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.127518 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.127567 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.128024 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57422: BlockingRecv error: recv got EOF from 127.0.0.1:57422 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 11:14:10.135869 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.134428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.7.90.125:44263)
0619 11:14:10.134558 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.134562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.135014 (+ 452us) server_negotiation.cc:407] Connection header received
0619 11:14:10.135189 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.135193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.135239 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.135290 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.135761 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57430: BlockingRecv error: recv got EOF from 127.0.0.1:57430 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 11:14:10.147650 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.146192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.7.90.125:44263)
0619 11:14:10.146289 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.146293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.146734 (+ 441us) server_negotiation.cc:407] Connection header received
0619 11:14:10.146943 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.146946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.147017 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.147070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.147559 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57442: BlockingRecv error: recv got EOF from 127.0.0.1:57442 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:14:10.168041 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.166373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.7.90.125:44263)
0619 11:14:10.166493 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.166498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.167054 (+ 556us) server_negotiation.cc:407] Connection header received
0619 11:14:10.167258 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.167262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.167315 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.167370 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.167934 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57444: BlockingRecv error: recv got EOF from 127.0.0.1:57444 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 11:14:10.204496 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.203212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.7.90.125:44263)
0619 11:14:10.203339 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.203344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.203589 (+ 245us) server_negotiation.cc:407] Connection header received
0619 11:14:10.203766 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.203770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.203845 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.203936 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.204378 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57446: BlockingRecv error: recv got EOF from 127.0.0.1:57446 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:14:10.272258 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.271010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.7.90.125:44263)
0619 11:14:10.271136 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.271139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.271331 (+ 192us) server_negotiation.cc:407] Connection header received
0619 11:14:10.271471 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.271474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.271523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.271618 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.272110 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57450: BlockingRecv error: recv got EOF from 127.0.0.1:57450 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 11:14:10.403476 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.402213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.7.90.125:44263)
0619 11:14:10.402388 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.402393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.402528 (+ 135us) server_negotiation.cc:407] Connection header received
0619 11:14:10.402725 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.402729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.402789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.402887 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.403343 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57456: BlockingRecv error: recv got EOF from 127.0.0.1:57456 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 11:14:10.662914 30302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:14:10.661760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.7.90.125:44263)
0619 11:14:10.661951 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:10.661955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:10.662034 (+ 79us) server_negotiation.cc:407] Connection header received
0619 11:14:10.662151 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:10.662153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:10.662206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:10.662289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:14:10.662713 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57468: BlockingRecv error: recv got EOF from 127.0.0.1:57468 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 11:14:10.988950 30391 catalog_manager.cc:1259] Loaded cluster ID: ff3f90b0e2d14bb689d1b08dd6c23106
I20260619 11:14:10.989053 30391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: loading cluster ID for follower catalog manager: success
I20260619 11:14:10.990839 30391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: acquiring CA information for follower catalog manager: success
I20260619 11:14:10.992079 30391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b0ec54ba91747538e668bcfb258c5c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:14:11.000216 30406 catalog_manager.cc:1259] Loaded cluster ID: ff3f90b0e2d14bb689d1b08dd6c23106
I20260619 11:14:11.000273 30406 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: loading cluster ID for follower catalog manager: success
I20260619 11:14:11.001850 30406 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: acquiring CA information for follower catalog manager: success
I20260619 11:14:11.002923 30406 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 523866ee61a5404cb3cbae78c45a77ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:14:11.180855 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:11.176928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.7.90.125:44263)
0619 11:14:11.177188 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:11.177192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:11.177261 (+ 69us) server_negotiation.cc:407] Connection header received
0619 11:14:11.177389 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:11.177391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:11.177512 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:11.177641 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:11.177860 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.178389 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.179293 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.179640 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.179717 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:11.180060 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:11.180496 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:11.180584 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:11.180688 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 11:14:11.180733 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260619 11:14:11.187757 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:11.184240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.7.90.124:37895)
0619 11:14:11.184474 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:11.184478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:11.184491 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:14:11.184614 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:11.184616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:11.184728 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:11.184814 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:11.184985 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.185470 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.186365 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.186599 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.186705 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:11.187015 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:11.187388 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:11.187517 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:11.187560 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 11:14:11.187608 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260619 11:14:11.194445 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:11.190631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.7.90.125:44263)
0619 11:14:11.190782 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:11.190786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:11.190929 (+ 143us) server_negotiation.cc:407] Connection header received
0619 11:14:11.191072 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:11.191075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:11.191124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:11.191199 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:11.191450 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.192150 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.193276 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.193573 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.193674 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:11.193935 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:11.194031 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:11.194095 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:11.194227 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 11:14:11.194273 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:14:11.195385 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:11.190886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.7.90.124:37895)
0619 11:14:11.191026 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:11.191031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:11.191408 (+ 377us) server_negotiation.cc:407] Connection header received
0619 11:14:11.191566 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:11.191569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:11.191614 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:11.191685 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:11.192014 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.192718 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.194085 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.194302 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.194441 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:11.194836 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:11.194919 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:11.194988 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:11.195197 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 11:14:11.195248 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:14:11.195713 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:14:11.190566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.7.90.126:35525)
0619 11:14:11.190795 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 11:14:11.190799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:14:11.191372 (+ 573us) server_negotiation.cc:407] Connection header received
0619 11:14:11.191580 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:14:11.191583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:14:11.191715 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:14:11.191827 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:14:11.192107 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.192745 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.194028 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:14:11.194370 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:14:11.194477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:14:11.194900 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:14:11.195259 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:14:11.195340 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:14:11.195504 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 11:14:11.195567 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260619 11:14:11.198462 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 30154
I20260619 11:14:11.204312 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 30229
I20260619 11:14:11.209904 7529 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7DZYiV/build/debug/bin/kudu with pid 30303
2026-06-19T11:14:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3019 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6063 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275356 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST